WKV-sharing embraced random shuffle RWKV high-order modeling for pan-sharpening
man zhou, Xuanhua He, Danfeng Hong, Bo Huang
Pan-sharpening aims to generate a spatially and spectrally enriched multi-spectral image by integrating complementarycross-modality information from low-resolution multi-spectral image and texture-rich panchromatic counterpart. In this work, we propose aWKV-sharing embraced random shuffle RWKV high-order modeling paradigm for pan-sharpening from Bayesian perspective, coupled with random weight manifold distribution training strategy derived from Functional theory to regularize the solution space adhering to thefollowing principles: 1) Random-shuffle RWKV. Recently, the Vision RWKV model, with its inherent linear complexity in global modeling,has inspired us to explore its untapped potential in pan-sharpening tasks. However, its attention mechanism, relying on a recurrentbidirectional scanning strategy, suffers from biased effects and demands significant processing time. To address this, we propose a novelBayesian-inspired scanning strategy called Random Shuffle, complemented by a theoretically-sound inverse shuffle to preserveinformation coordination invariance, effectively eliminating biases associated with fixed sequence scanning. The Random Shuffleapproach mitigates preconceptions in global 2D dependencies in mathematical expectation, providing the model with an unbiased prior.In line with similar spirit of Dropout, we introduce a testing methodology based on Monte Carlo averaging to ensure the model’s outputaligns more closely with expected results. 2) WKV-sharing high-order. Regarding KV’s attention score calculation in spatial mixer of RWKV, we leverage WKV-sharing mechanism to transfer KV activations across RWKV layers, achieving lower latency and improved trainability, and revisit the channel mixer in RWKV, originally a first-order weighting function, and redevelop its high-order potential by sharing the gate mechanism across RWKV layer. Comprehensive experiments across pan-sharpening benchmarks demonstrate our model’s effectiveness, consistently outperforming state-of-the-art alternatives