Deep Residual Echo State Networks: exploring residual orthogonal
  connections in untrained Recurrent Neural Networks
Echo State Networks (ESNs) are a particular type of untrained Recurrent
Neural Networks (RNNs) within the Reservoir Computing (RC) framework, popular
for their fast and efficient learning. However, traditional ESNs often struggle
with long-term information processing. In this paper, we introduce a novel
class of deep untrained RNNs based on temporal residual connections, called
Deep Residual Echo State Networks (DeepResESNs). We show that leveraging a
hierarchy of untrained residual recurrent layers significantly boosts memory
capacity and long-term temporal modeling. For the temporal residual
connections, we consider different orthogonal configurations, including
randomly generated and fixed-structure configurations, and we study their
effect on network dynamics. A thorough mathematical analysis outlines necessary
and sufficient conditions to ensure stable dynamics within DeepResESN. Our
experiments on a variety of time series tasks showcase the advantages of the
proposed approach over traditional shallow and deep RC.
68b59b05dc94ad77a7bfcffc
Matteo Pinna
claimed_verified
2025-09-01T13:46:18.949Z
6898c8c3e9e1c6671e64a8ff
https://cdn-avatars.huggingface.co/v1/production/uploads/6898c8c3e9e1c6671e64a8ff/NLRxGoqH036cq5x55wvk6.jpeg
Matteo Pinna
user
nennomp
68b59b05dc94ad77a7bfcffd
Andrea Ceni
68b59b05dc94ad77a7bfcffe
Claudio Gallicchio
Deep Residual Echo State Networks: exploring residual orthogonal
  connections in untrained Recurrent Neural Networks
Echo State Networks (ESNs) are a particular type of untrained Recurrent
Neural Networks (RNNs) within the Reservoir Computing (RC) framework, popular
for their fast and efficient learning. However, traditional ESNs often struggle
with long-term information processing. In this paper, we introduce a novel
class of deep untrained RNNs based on temporal residual connections, called
Deep Residual Echo State Networks (DeepResESNs). We show that leveraging a
hierarchy of untrained residual recurrent layers significantly boosts memory
capacity and long-term temporal modeling. For the temporal residual
connections, we consider different orthogonal configurations, including
randomly generated and fixed-structure configurations, and we study their
effect on network dynamics. A thorough mathematical analysis outlines necessary
and sufficient conditions to ensure stable dynamics within DeepResESN. Our
experiments on a variety of time series tasks showcase the advantages of the
proposed approach over traditional shallow and deep RC.
Deep Residual Echo State Networks (DeepResESNs) enhance long-term temporal modeling and memory capacity through hierarchical untrained residual layers, outperforming traditional shallow and deep reservoir computing methods.
Echo State Networks Recurrent Neural Networks Reservoir Computing temporal residual connections Deep Residual Echo State Networks DeepResESNs network dynamics time series tasks