\section{Introduction}
\IEEEPARstart{S}{imultaneous} localization and mapping (SLAM) is one of the most fundamental tasks in the field of computer vision and robotics, with applications ranging from augmented reality (AR), virtual reality (VR) to autonomous driving. In AR applications, SLAM is often used to provide accurate localization to facilitate users to place virtual objects~\cite{Ye2021SuperPlane}, while the dense reconstruction is increasingly needed for better interaction with the environment. Monocular dense SLAM~\cite{deepfactors, tateno2017cnn} has received much attention due to the simplicity of monocular video acquisition, yet it is a much more difficult task compared to RGB-D SLAM~\cite{wang2021DymSLAM, Li2022DGS-SLAM, yu2018dsslam, dai2020rgb, henein2020dynamic}.

\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{pictures/dual-flow_representation.png}
  \caption{\textbf{Dual-Flow Representation.} The estimated optical flow is decomposed into a static flow caused by the camera poses and a dynamic flow caused by the dynamic object's motion.}
  \label{fig: dual-flow representation}
  \vspace{-1em}
\end{figure}

Impressive progress has been seen in geometry-based~\cite{Engel2014lsd, mur2015orb, forster2014svo, engel2017direct}, learning-based \cite{Zhou2017SfMLearner, wang2018end, wang2019improving}, and hybrid approaches \cite{wang2017non, wang2017noniterative, brahmbhatt2018geometry}. However, developing robust and reliable SLAM methods for real-world AR applications is still a challenging problem, especially in dynamic scenarios. To meet the challenges of such dynamic environments, some approaches~\cite{bescos2018dynaslam, xiao2019dynamic} filter out dynamic objects in advance by introducing information such as semantics, e.g., using Mask R-CNN~\cite{He_2017_ICCV} to segment out potentially moving vehicles and pedestrians, and then run monocular SLAM systems such as ORB-SLAM~\cite{mur2015orb}. However, due to the slow running speed and high memory consumption of Mask-RCNN, it is difficult to meet the real-time requirement of SLAM. What's more, there are limits in practical applications, for not all dynamic objects exist in the detector's training data and may lead to catastrophic system failure~\cite{wang2021DymSLAM}.
Besides, simply throwing away dynamic information and only constructing sparse feature maps makes such SLAM methods incapable of handling tasks like planning and interaction.
Some approaches \cite{yang2019cubeslam, huang2020clustervo} combine multi-object detection and SLAM, making it possible to add target-aware constraints to mitigate the interference of dynamic objects, but the generalization of such SLAM systems is limited as well due to the detector.

Recently, a learning-based dense SLAM system, DROID-SLAM~\cite{teed2021droid}, has been proposed, which demonstrates better accuracy and robustness than traditional methods. However, we find that it has relatively large errors in the pose estimation in some stronger dynamic scenarios such as sequence 09 of KITTI~\cite{geiger2013vision}. DROID-SLAM~\cite{teed2021droid} presents a dense bundle adjustment layer to iteratively update the residual of inverse depths and camera poses using the estimated optical flow and weights, while the presence of dynamic objects may lead to ambiguity in optical flow estimation if the system failed to discover them. For this reason, we propose a novel scene motion representation, named dual-flow, that decomposes the estimated optical flow into a static flow caused by the camera poses and a dynamic flow of the dynamic object's own motion, as shown in Fig. \ref{fig: dual-flow representation}. Such a representation largely mimics the way humans perceive the real world~\cite{villacorta2021static}. In addition, DROID-SLAM introduces massive supervision for optical flow, depth, and pose estimation during training, which severely restricts the possibility of finetuning in new scenes if it performs poorly in new scenes. On the contrary, we explore to train our dual-flow-based dynamic SLAM in a self-supervised way.

Our insight is that for the vast majority of scenes, dynamic objects may not be dominant and their motion can be decomposed by the dominant static objects. The representation based on scene motion decomposition has natural advantages: 1) we can easily optimize static fields in a similar way as DROID-SLAM does for the poses and depths.  2) For dynamic fields, we can obtain consistent luminosity by warping the current frame to adjacent frames.  3) This holds promise for obtaining a self-supervised network model, which is also more interpretable.
Based on these observations, we propose a dynamic update module for DeFlowSLAM, in which the dual-flow representation is cleverly embedded, shown in Fig. ~\ref{fig:pipeline}, and detailed in Sec \ref{sec:overview}. Directly warping the dynamic object field to the neighbor images may result in occlusion, shown in Fig. \ref{fig:maskagg}, we introduce a dynamic mask aggregation operator in the dynamic update module, named Mask-Agg to remove the incorrect alignment effect. Specifically, the Mask-Agg operator iteratively updates the residuals of the dynamic mask via convolutional gated recurrent unit~(ConvGRU). The final dynamic mask can be obtained by summing the aggregated dynamic mask residuals with the original mask. The obtained dynamic mask will be combined with the estimated weights and fed into the dense bundle adjustment (DBA) Layer to optimize the residuals of pose and depth, shown in Fig. \ref{fig:update module}.

We first verify our hypothesis on a highly dynamic dataset, VKITTI2~\cite{cabon2020virtual}, where ablation experiments demonstrate that a dual-flow representation can achieve better performance. In addition, DeFlowSLAM can make full use of information from all pixels to solve the camera pose, resulting in gains compared to potential approaches that filter dynamic objects by masking. Applying our knowledge to the challenging SLAM dataset of TartanAir, the trained model exhibits good generalization and achieves comparable accuracy to DROID-SLAM in both static and slightly dynamic scenes. While in highly dynamic scenes, our method significantly outperforms DROID-SLAM, and even the error in pose estimation is sharply reduced to half of DROID-SLAM. Then, we simply extended it to stereo and RGB-D datasets, showing the high scalability of the system. In AR applications, DeFlowSLAM shows relatively accurate pose estimation results, as shown in Fig. \ref{fig:AR demo}.
 In summary, our contributions are three-fold:
\begin{itemize}
    \item We propose a novel dual-flow scene motion representation that decomposes the optical flow into a static flow field and a dynamic flow field.
    \item We build a dynamic dense SLAM, DeFlowSLAM, which outperforms state-of-the-art methods in dynamic scenes.
    \item We propose a self-supervised training method instead of the strong supervision in DROID-SLAM.

\end{itemize}

\begin{figure*}[t]
    \centering
    \includegraphics[width=.95\linewidth]{pictures/DeFlowSLAM_framework.pdf}
    \vspace{-1em}
    \caption{\textbf{DeFlowSLAM Overview.} DeFlowSLAM takes the image sequence as input, extracts features to construct a correlation volume, and feeds it with the initial static flow, optical flow, and dynamic mask into the dynamic update module to iteratively optimize the residual of the pose, inverse depth, static optical flow and dynamic optical flow, and finally outputs the estimation of the camera pose and 3D structure. The optimization process is performed by creating a covisibility graph and updating the existing covisibility graph.}
    \label{fig:pipeline}
    \vspace{-1em}
\end{figure*}

\vspace{-1em}

\section{Related Work}
\subsection{Scene Motion Decomposition}
Scene motion estimation has received increasing attention for 3D perception. It is the task of obtaining the 3D structure and 3D motion of dynamic scenes. Many scene motion estimation approaches have been proposed recently based on different types of input data, such as 3D point clouds~\cite{liu:2019:flownet3d, HPLFlowNet}, stereo images~\cite{huguet2007variational, vogel2013piecewise, zhang20013d}, or RGB-D images~\cite{hornacek2014sphereflow, hadfield2011kinecting, Lv18eccv, quiroga2014dense}.  Monocular scene flow estimation is a highly ill-posed problem and comparatively few approaches have been suggested so far~\cite{brickwedde2019mono}. Here, we focus on monocular videos and the scene motion estimation can be regarded as optical flow~\cite{dosovitskiy2015flownet, ilg2017flownet, sun2018pwc, ranjan2019competitive}. RAFT~\cite{teed2020raft} presents a Recurrent All-pairs Field Transforms network for optical flow estimation, which does not consider the scene motion decomposition. While EffiScene~\cite{Jiao_2021_CVPR} presents an unsupervised scene flow estimation method by jointly learning four low-level vision sub-tasks: optical flow, depth, camera pose, and motion segmentation.
Different from the forward and backward flow proposed by DF-VO~\cite{zhan2021dfvo}, we leverage the property \cite{Baur2021ICCV} of the optical flow, where it inherently separates every pixel into multiple moving agents and a large class of points following a single rigid sensor motion and try to decompose the optical flow to the static field caused by the camera pose and dynamic field caused by the objects' own motion.
\vspace{-1em}
\subsection{Dynamic SLAM}
SLAM is a fundamental capability of many intelligent systems to provide accurate pose estimation and mapping. Humans live in a dynamic environment, intelligent systems should also have the ability to deal with dynamic environments, which are required to recognize the dynamic contents from the static environments. Traditional approaches largely filter out the interference of dynamic objects by introducing a prior~\cite{tan2013robust} or RANSAC methods~\cite{huang2020dual}. Some recent approaches try to use segmentation to filter out potential dynamic objects~\cite{vincent2020dynamic, bescos2018dynaslam, xiao2019dynamic} and then run the SLAM system, or unify object detection and SLAM into a multi-task system~\cite{huang2020clustervo, nair2020multi, Ye2022PVO} or add the object constraint to the SLAM system~\cite{yang2019cubeslam, strecke2019fusion, brasch2018semantic}. Unlike existing approaches, this paper uses a learning-based approach to identify dynamic fields at the pixel level, which does not require explicit supervision of object detection and better simulates the way humans see the world.

\vspace{-1em}

\subsection{Dense Reconstruction}
Reconstructing dense volumetric scenes is an important task in robotics with many applications such as city modeling, augmented reality navigation~\cite{Wasenmller2016AugmentedR3}, and cultural heritage preservation~\cite{10.1145/2770877}. Most of the existing systems~\cite{reijgwart2019voxgraph, oleynikova2017voxblox, millane2018c, slavcheva2017killingfusion, bozic2020deepdeform} need to use depth or lidar information for dense reconstruction, this paper does not have this limitation, and can support data formats such as monocular video, stereo, and RGB-D, and can maintain a good reconstruction effect in the dynamic environment. To improve the reconstruction performance, some approaches propose collaborative reconstruction by multiple robots~\cite{li2017corb, schmuck2019ccm, Liu2021Coxgraph}. As an example, Coxgraph~\cite{Liu2021Coxgraph} extends the single-robot reconstruction system, Voxgraph~\cite{reijgwart2019voxgraph}, to a multi-robot dense reconstruction system, and additionally adds more constraints to ensure low-error reconstruction. Our system is able to maintain robust dense reconstruction with less constraint. Some methods focus on dense depth reconstruction~\cite{wimbauer2020monorec, ranftl2016dense, russell2014video}, while our DeFlowSLAM focuses on pose estimation although it can produce the intermediate depth and flow.

\vspace{-1em}
\section{Methodology}
\label{sec:overview}
Fig. \ref{fig:pipeline} shows an overview of our novel dual-flow-based SLAM method, DeFlowSLAM, which takes a sequence of images as input and outputs the estimation of the camera pose and the 3D map of the environments. DeFlowSLAM has an end-to-end differentiable architecture that combines the strengths of both classical approaches and deep networks. It can robustly cope with challenging scenarios such as dynamic scenes, thanks to our proposed dual-flow representation, iterative dynamic update module, and factor graph optimization based on co-visibility between frames. Specifically, unlike DROID-SLAM~\cite{teed2021droid}, which iteratively updates camera pose and depth, we additionally update a dynamic mask and a dynamic flow. And each update of camera poses in DeFlowSLAM is performed by the optimization of the estimated static flow field rather than the optical flow in DROID-SLAM. Next, we will first review DROID-SLAM for understanding and then elaborate on the details of our approach.

\vspace{-1em}

\subsection{Preliminaries: DROID-SLAM}
DROID-SLAM~\cite{teed2021droid} operates on a sequence of images $\{\mathbf{I}_t\}_{t=0}^N$. For each image $t$, it maintains two state variables: camera pose $\mathbf{G}_t \in SE(3)$ and inverse depth $\mathbf{d}_t \in \mathbb{R}_{+}^{H\times W}$, which get iteratively updated during inference as new frames are processed. DROID-SLAM~\cite{teed2021droid} adopts a frame graph $\mathcal{G} = (\mathcal{V}, \mathcal{E})$ to represent co-visibility between frames. The nodes are input images and the edge $(i, j) \in \mathcal{E}$ implies that the images ${I}_i$ and ${I}_j$ have overlapped views.

\noindent\subsubsection{Feature Extraction and Correlation}
Following RAFT \cite{teed2020raft}, the input images are first fed to the feature extraction module, and the relationship between the two images will be computed.

\noindent\textbf{Feature Extraction.}
First, each of the input images is processed by an image encoder, which consists of six residual blocks and three downsampling layers. The image encoder produces a dense image feature map, which is 1/8 of the original resolution, and feature maps from pairs of input images are used for later correlation volume construction.

\noindent\textbf{Correlation Pyramid.}
Given pairs of image features for each edge $(i, j) \in \mathcal{E}$ in the frame graph $\mathcal{G}$, DROID-SLAM\cite{teed2021droid} generates the correlation volume by taking the dot product between all-pairs of image feature vectors in $\mathbf{f}_\theta(I_i)_{u_i v_i}$ and $\mathbf{f}_\theta(I_j)_{u_j v_j}$, which has 4 dimensions.
It then computes the correlation volume $\mathbf{C}^{ij} \in \mathbb{R}^{H\times W\times H\times W}$ with each element computed as
\begin{equation}
    C_{u_i v_i u_j v_j}^{ij} = \langle \mathbf{f}_\theta(I_i)_{u_i v_i},\  \mathbf{f}_\theta(I_j)_{u_j v_j} \rangle.
\end{equation}
where $u_i,v_i,u_j,v_j$ are the pixel coordinates for image $I_i, I_j$ respectively and $< . >$ stands for the dot product.
The last two dimensions of the correlation matrix are fed to the average pooling layers with four different kernel sizes (1,2,4,8), forming a 4-level correlation pyramid.

\noindent\textbf{Correlation Lookup.}
DROID-SLAM \cite{teed2021droid} defines a correlation lookup function that uses a coordinate grid with radius $r$ to index the correlation volume $L_r:
\mathbb{R}^{H\times W\times H\times W} \times \mathbb{R}^{H\times W \times 2} \mapsto \mathbb{R}^{H\times W \times (r+1)^2}$.
This function takes an $H\times W$ grid of coordinates as input, which is the dimension of the optical flow field.
Then it retrieves the relative values from the correlation volume by using a bi-linear interpolation. The correlation lookup function is applied to every correlation volume matrices in the pyramid. The final output feature is combined by concatenating the lookup results at each level.

\noindent\subsubsection{Update Module}
DROID-SLAM~\cite{teed2021droid} proposes an update operator, a $3\times3$ convolutional GRU which updates the hidden state$\mathbf{h}$, and additionally produces a pose update, $\Delta \boldsymbol \xi^{(k)}$, and depth update, $\Delta \mathbf{d}^{(k)}$. In the update module, the pose and depth updates are applied to the current depth and pose estimates through retraction on the SE3 manifold and vector addition respectively
\begin{equation}
    \mathbf{G}^{(k+1)} = \Exp(\Delta \boldsymbol \xi^{(k)}) \circ \mathbf{G}^{(k)}, \qquad
    \mathbf{d}^{(k+1)} = \Delta \mathbf{d}^{(k)} + \mathbf{d}^{(k)}.
\end{equation}
The update operator iteratively produces a sequence of poses and depths, with the expectation of converging to a fixed point $\{\mathbf{G}^{(k)}\} \rightarrow \mathbf{G}^*$, $\{\mathbf{d}^{(k)}\} \rightarrow \mathbf{d}^*$, reflecting the true reconstruction.

\begin{figure*}[t]
    \centering
    \includegraphics[width=0.95\linewidth]{pictures/dynamic_update_module.pdf}

    \caption{\textbf{Dynamic Update Module.} The correlation feature of the static optical flow is looked up by correlation volumes. The obtained features will be fed into two convolutional layers together with the optical flow and dynamic mask, resulting in intermediate features. These features will be fed to ConvGRU, followed by two convolution layers with dynamic mask residual and confidence. The iterative dynamic mask residual plus the original mask to obtain the new dynamic mask termed Mask-Agg Operator. In addition, static flow revision and dynamic flow are obtained from the static and dynamic convolution layers. The static revision flow plus the original static flow is fed into the DBA layer that combines the dynamic mask and confidence to optimize the depth and pose. Finally, the new static and dynamic flows from the dynamic convolutional layer are summed to the optical flow.}
    \label{fig:update module}
    \vspace{-1em}
\end{figure*}

\vspace{-1em}
\subsection{Dual-Flow Representation}
The core concept of the proposed dynamic SLAM network is the dual-flow representation and the self-supervised training scheme. In contrast to DROID-SLAM \cite{teed2021droid}, which uses optical flow as an intermediate motion representation, we propose a novel scene motion representation by decomposing optical flow into a static flow caused by camera motion and a dynamic flow caused by the motion of dynamic objects themselves, shown in Fig. \ref{fig: dual-flow representation}. Such a representation can distinguish between static and dynamic object motions, thus having better interpretability and making the network traceable during the training process. We can directly use the dense static flow to estimate camera motion without masking out the pixels belonging to dynamic objects.
The optical flow $\mathbf{F_{o}}_t \in \mathbb{R}^{H\times W\times 2}$, static flow $\mathbf{F_{s}}_t \in \mathbb{R}^{H\times W\times 2}$ and dynamic flow $\mathbf{F_{d}}_t \in \mathbb{R}^{H\times W\times 2}$ are a set of vectors, where the static flow plus the dynamic flow equals the optical flow:
\begin{equation}
 \mathbf{F_{o}}_t = \mathbf{F_{s}}_t + \mathbf{F_{d}}_t.
 \label{equ: flow equ}
\end{equation}

Our network operates on a sequence of images $\{\mathbf{I}_t\}_{t=0}^N$. As new frames being processed, the set of camera poses $\{\mathbf{G}_t\}_{t=0}^N \in SE(3)$, inverse depths $\{\mathbf{d}_t\}_{t=0}^N \in \mathbb{R}_{+}^{H\times W}$, static flows $\{\mathbf{F_{s}}_t\}_{t=0}^N \in \mathbb{R}^{H\times W\times 2}$ , dynamic flows $\{\mathbf{F_{d}}_t\}_{t=0}^N\in \mathbb{R}^{H\times W\times 2}$ and the binary dynamic masks $\{\mathbf{M_{d}}_t\}_{t=0}^N \in \mathbb{R}_{+}^{H\times W \times 2 }$ are update iteratively.
In $\mathbf{M_{d}}_t$, we let 0 indicate dynamic while 1 indicate static.
As a reminder in this paper, when we refer to depth, please note that we are using the inverse depth parameterization.

Similar to DROID-SLAM~\cite{teed2021droid}, we also adopt a frame graph $\mathcal{G} = (\mathcal{V}, \mathcal{E})$ to represent co-visibility between frames.
For example, as shown in the co-visibity graph optimization in Fig. \ref{fig:pipeline}, the white nodes indicate each image, and the edges indicate two images with a co-visibility relationship, where the red points indicate image ${I}_i$ and image ${I}_j$, and the blue edges indicate the co-visibility relationship in the frame graph.
The frame graphs are built dynamically during the training and inference. After each pose or depth update with the revision static flow field,
we can update the frame graph with new co-visibilities.
When the camera returns to a previously visited place, a long-distance connection is added to the graph, and loop closure is performed.

\vspace{-1em}
\subsection{Dynamic Update Module}
Fig. \ref{fig:update module} shows the dynamic update module of our DeFlowSLAM.
The module is a $3\times3$ ConvGRU with a hidden state $\mathbf{h}$. Different from the update module in DROID-SLAM \cite{teed2021droid}, which works directly on the revision optical flow, our dynamic update module works on the decomposed static and dynamic flow fields respectively. We update the revision static flow field in a similar way to DROID-SLAM \cite{teed2021droid}, while for the dynamic flow field, we add it to the static flow field to obtain the optical flow, which is fed into the flow encoder as a new optimization term in the next iteration.
Each application of the operator updates the hidden state, and additionally produces a pose increment, depth increment, dynamic mask increment, and dynamic flow. The pose increments are applied to the current pose through retraction on the SE3 manifold,
\begin{equation}
    \mathbf{G}^{(k+1)} = \Exp(\Delta \boldsymbol \xi^{(k)}) \circ \mathbf{G}^{(k)}.
\end{equation}
While the depth and the dynamic mask increments are added to the current depth and dynamic mask respectively
\begin{equation}
    \Xi^{(k+1)} = \Delta \Xi^{(k)} + \Xi^{(k)}, \, \Xi \in \{\mathbf{d}, \mathbf{M_d}\}.
\end{equation}
where $\mathbf{F_d}^{(k+1)}$ is directly given a new value in every iteration. With the updated static flow $\mathbf{F_{s}}^{(k+1)}$ transformed from $\mathbf{G}^{(k+1)}$ and $\mathbf{d}^{(k+1)}$, the final optical flow can be computed using Equation \ref{equ: flow equ}.

Iterative operations of the dynamic update module produce a sequence of poses, depths, dynamic masks, dynamic flows, and complete optical flows with the expectation of converting to an optimal position, such as
$\{\mathbf{G}^{(k)}\} \rightarrow \mathbf{G}^*$, $\{\mathbf{d}^{(k)}\} \rightarrow \mathbf{d}^*$, $\{\mathbf{M_{d}}^{(k)}\} \rightarrow \mathbf{M_{d}}^*$, $\{\mathbf{F_{d}}^{(k)}\} \rightarrow \mathbf{F_{d}}^*$, $\{\mathbf{F_{o}}^{(k)}\} \rightarrow \mathbf{F_{o}}^*$,
representing the true reconstruction.

\subsubsection{Correspondence}
At the beginning of each iteration, we use the current pose and depth estimates to search for correspondence. Given a grid of pixel coordinates, $\mathbf{p}_i \in \mathbb{R}^{H \times W \times 2}$ in frame $i$, we compute the dense correspondence field $\mathbf{p}_{ij}$
\begin{equation}
    \mathbf{p}_{ij} = \Pi_c(\mathbf{G}_{ij} \circ \Pi_c^{-1}(\mathbf{p}_i, \mathbf{d}_i)),

    \  \mathbf{G}_{ij} = \mathbf{G}_j \circ \mathbf{G}_i^{-1}.
\end{equation}
for each edge $(i, j) \in \mathcal{E}$ in the frame graph. Here $\Pi_c$ is the camera model that maps a set of 3D points onto the image, and $\Pi_c^{-1}$ is the inverse projection function that maps the inverse depth map $\mathbf{d}$ and the coordinate grid $\mathbf{p}_i$ to the 3D point cloud. $\mathbf{p}_{ij}$ represents the coordinates of pixel $\mathbf{p}_i$ mapped to $j$ frames using the current pose and depth estimates.

\subsubsection{Inputs}
We use the dense correspondence field to index the correlation volumes. For each edge of $(i,j) \in \mathcal{E}$, we use $\mathbf{p}_{ij}$ to lookup from the correlation volume $\mathbf{C}^{ij}$ to retrieve the correlation features. In addition, we use the dense correspondence field to derive the static optical flow caused by the camera motion, i.e., the difference between $\mathbf{p}_{ij} - \mathbf{p}_j$, which is $\mathbf{F_{s}}_{ij}$. For the initial dynamic mask $\mathbf{M_d}_{ij}$ and the initial dynamic flow $\mathbf{F_d}_{ij}$, we simply initialize them to zeros.

\subsubsection{Dynamic Update}
We extract the global context by averaging the hidden states over the spatial dimensions of the image and use this feature vector as an additional input to the ConvGRU.
ConvGRU generates an updated hidden state $\mathbf{h}^{(k+1)}$. Instead of directly predicting updates in depth or pose, we predict updates in the space of dense flow fields. We map the hidden states by two additional convolution layers to produce four outputs: (1) a revised static flow field $\mathbf{r_s}_{ij} \in \mathbb{R}^{H \times W \times 2}$, (2) an updated dynamic flow field $\mathbf{F_d}_{ij} \in \mathbb{R}^{H \times W \times 2}$,  (3) a correlation confidence map $\mathbf{w}_{ij} \in \mathbb{R}^{H \times W \times 2}$, (4) an updated dynamic mask increment field $\Delta \mathbf{M_d}_{ij} \in \mathbb{R}^{H \times W \times 2}$. The revision $\mathbf{r}_{ij}$ is a correction term for the network prediction, which is used to correct errors in the dense correspondence fields. We express the corrected correspondence as $\mathbf{p_s}_{ij}^* = \mathbf{r_s}_{ij} + \mathbf{p_s}_{ij}$.

We then pool the hidden state's overall features that share the same source view $i$  and predict the pixel-level damping factor $\lambda$. We use the softplus operator to ensure that the damping term is positive. In addition, we use pooled features to predict an 8x8 mask that can be used to upsample the inverse depth estimation.

\subsubsection{Dense Bundle Adjustment Layer (DBA)}
The dense bundle adjustment layer (DBA) maps a set of stream revisions to a set of pose and pixel depth updates. We define the cost function on the whole frame diagram
\begin{equation}
    \mathbf E(\mathbf{G}', \mathbf{d}') = \sum_{(i,j) \in \mathcal{E}} \norm{\mathbf{p_s}_{ij}^* - \Pi_c(\mathbf{G}'_{ij} \circ \Pi_c^{-1}(\mathbf{p}_i, \mathbf{d}'_i)) }_{\Sigma_{ij}}^2
    \label{eqn:objective},
\end{equation}
\begin{equation}
\Sigma_{ij} = \diag \mathbf{w_d}_{ij},
\end{equation}
\begin{equation}
    \mathbf{w_d}_{ij}=\operatorname{sigmoid}(\mathbf{w}_{ij}+(1-\mathbf{M_d}_{ij})\cdot\eta).
\end{equation}
where $\eta$ is set as 10. And $\norm{\cdot}_{\Sigma}$ is the Mahalanobis distance, which weights the error term according to the combined confidence $\mathbf{w_d}_{ij}$. The combined confidence makes sure that the optimization considers all points.
The addition of these points can effectively improve the accuracy of BA operation in dynamic scenes. Equation \ref{eqn:objective} shows that we want an updated pose $\mathbf{G}'$ and depth $\mathbf{d}'$ such that the reprojected points match the modified correspondence $\mathbf{p_s}_{ij}^*$ predicted by the update operator.
\begin{figure}[t]
  \centering
  \includegraphics[width=\linewidth]{pictures/maskagg.png}
  \caption{\textbf{Mask-Agg Illustration}. If there are dynamic objects in the scene, directly matching pixels using static flow will cause erroneous results, making photometric loss invalid. Using the aggregated mask predicted by the network, we can filter out these invalid pixels (e.g. the pink mask) in geometry photometric loss, which is shown in the right bottom picture.}
  \label{fig:maskagg}
  \vspace{-1em}
\end{figure}
We use a local parameterization to linearize Equation \ref{eqn:objective} and use the Gauss-Newton algorithm to solve for the update ($\Delta \boldsymbol\xi$, $\Delta \mathbf d$). Since each term in Equation \ref{eqn:objective} contains only a single depth variable, the Hessian matrix has a block diagonal structure. Separating the camera pose and depth variables allows the system to be solved efficiently using Schur's complement and adding the pixel damping factor $\lambda$ to the depth block.
The DBA layer is implemented as part of the computational graph and backpropagation is performed through this layer during training.

\vspace{-1em}
\subsection{Loss Function}
In this section, we elaborate on the self-supervised training scheme for DeFlowSLAM in detail.

\subsubsection{Geometry Photometric Loss}
To supervise the geometric predictions using images, we introduce a photometric reprojection loss to guide the network's optimization. We refer to \cite{2020Digging} to build our photometric loss function.
Given the predicted pose $G_{ij}$ and the predicted depth $\mathbf{\hat{d}}_i$, we can get the corresponding coordinates of pixels in image $I_i$ in image $I_j$. We then use bi-linear sampling to sample the image $I_j$, getting a new-sampled image $I_{j \rightarrow i}$.
\begin{equation}
    I_{j \rightarrow i}=I_{j}\left\langle\Pi_c(\mathbf{G}_{ij} \circ \Pi_c^{-1}(\mathbf{p}_i, \mathbf{\hat{{d}}}_i))\right\rangle.
\end{equation}
Then we can use photometric loss on source image $I_i$ and new image $I_{j \rightarrow i}$.
\begin{equation}
    \mathcal{L}_{\text {geo\_ph}}=\frac{1}{N}\sum_{ij} p e\left(I_{i}, I_{j \rightarrow i}\right).
\end{equation}
We introduce $L_1$ loss and $SSIM$\cite{2004Image} loss to form our geometry photometric loss, setting $\alpha=0.85$.
\begin{equation}
    p e\left(I_{a}, I_{b}\right)=\frac{\alpha}{2}\left(1-\operatorname{SSIM}\left(I_{a}, I_{b}\right)\right)+(1-\alpha)\left\|I_{a}-I_{b}\right\|_{1}.
\end{equation}

\subsubsection{Mask Aggregation Operator for Geometry}
In the self-supervised training policy, when using the photometric loss to supervise camera poses and depths, the direct use of the dual-flow representation may result in pixel mismatches due to the object's motion, making the geometry photometric loss less accurate. So we propose a mask aggregation operator, i.e. Mask-Agg, to filter out the wrong pixel matches to enhance our dual-flow representation, as shown in Fig. \ref{fig:maskagg}.
The aggregated mask $\mathbf{M_{d_i}^{Agg}}$ for frame $I_i$ is
computed by gathering the dynamic masks estimated for all the frames which are connected to frame
$I_i$ in the frame graph.
Then, the final geometry photometric loss function is
\begin{equation}
     \mathcal{L}_{\text {geo\_ph}}=\frac{1}{N^{'}}\sum_{ij} p e\left(I_{i}, I_{j \rightarrow i}\right) \cdot \mathbf{M_{d_i}^{Agg}}.
\end{equation}
where $N^{'}$ means the count of pixels whose $\mathbf{M_{d_i}^{Agg}}$ value is 1. Specifically, the Mask-Agg operator iteratively updates the residuals of the dynamic mask via ConvGRU. The final dynamic mask can be obtained by summing the aggregated dynamic mask residuals with the original old mask.
Tab. \ref{table:vkitti ablation study} shows the Mask-Agg module helps filter out the ambiguous matches in self-supervised training, obtaining better results.

\subsubsection{Optical Flow Photometric Loss}
The geometry photometric loss is used to supervise the static flow caused by camera motion. So we also introduce the optical flow photometric loss to supervise the complete scene motion, including camera motion and objects' motion. Through the update module, we can get the optical flow results $\mathbf{F_o}_{ij}$ by adding static flow and dynamic flow. Similar to the geometry photometric loss, we use $\mathbf{F_o}_{ij}$ to generate corresponding coordinates between images.
\begin{equation}
    I_{j \rightarrow i}=I_{j}\left\langle \mathbf{F_o}_{ij}+ \mathbf{p_{ij}}\right\rangle.
\end{equation}
Then we still use bi-linear sampling to sample from the source image, evaluate their photometric errors:
\begin{equation}
    \mathcal{L}_{\text {flow\_ph}}=\sum_{ij} p e\left(I_{i}, I_{j \rightarrow i}\right).
\end{equation}
where the $pe$ function here is just $L_1$ loss:
\begin{equation}
    p e\left(I_{a}, I_{b}\right)=\left\|I_{a}-I_{b}\right\|_{1}.
\end{equation}

\subsubsection{Supervised Mask Loss}
When dynamic mask labels are available, we can directly supervise our predicted masks using a simple cross-entropy classification loss:
\begin{equation}
    \mathcal{L}_{\text{gt\_mask}} = -\frac{1}{\left|\mathcal N\right|} \sum_{\mathbf{p}_{i} \in \mathcal N} \mathbf{M}_{i} \log \mathbf{\hat M}_i + \left(1-\mathbf{M}_{i}\right) \log \left(1-\mathbf{\hat M}_i\right).
\end{equation}
where $\mathbf{M}_{i}$ is the ground truth mask labels, $\mathbf{\hat M}_i$ is the predicted masks.

\subsubsection{Artificial Mask Loss}
When the dynamic mask labels are not available, we design a method to artificially build referred masks for self-supervised dynamic mask learning. Referred to SLIM~\cite{Baur2021ICCV}, we propose the artificial mask loss to achieve this goal. Using the camera pose, depth, and optical flow we have already gotten, we can first infer the target coordinate of the pixel $p$ following the equations below:
\begin{equation}
    \mathbf{p}_{\text {cam}} =\Pi_c(\mathbf{G}_{ij} \circ \Pi_c^{-1}(\mathbf{p}_i, \mathbf{\hat{{d}}}_i)), \\
    \mathbf{p}_{\text {flow}} =\mathbf{p}_i+ \mathbf{\hat F_o{}}_{ij}.
\end{equation}
where $\mathbf{p}_{\text{cam}}$ is the target coordinate calculated by projection, $\mathbf{p}_{\text{flow}}$ is the target coordinate calculated by optical flow.
Then we use the difference between these two coordinates to form our artificial mask labels
\begin{equation}
    \mathbf{M^{art}_{i}}=[\left\|\mathbf{p}_{\text {cam}}-\mathbf{p}_{\text {flow}}\right\|_{2}\le \mu],
\end{equation}
where $\mu$ is set as 0.5. This artificial mask label has the same format compared with ground truth mask labels, so our final loss function should be
\begin{align}
\mathcal{L}_{\text{art\_mask}} = &-\frac{1}{\left|\mathcal N\right|} \sum_{\mathbf{p}_{i} \in \mathcal N} \mathbf{M^{art}_{i}} \log \mathbf{\hat M}_i + \nonumber\\
    &\left(1-\mathbf{M^{art}_{i}}\right) \log \left(1-\mathbf{\hat M}_i\right).
\end{align}
\begin{figure}[t]
  \centering
  \includegraphics[width=\linewidth]{pictures/slam.pdf}

  \caption{\textbf{Qualitative Results of DeFlowSLAM}. DeFlowSLAM can generalize to new datasets. We show results from ETH3D, TUM RGB-D, ScanNet and Virtual KITTI2.}
  \label{fig:slam reconstruction}
  \vspace{-1em}
\end{figure}

\subsubsection{Final Loss Function}
We use $\mathcal{L}_{\text {geo\_ph}}$, $\mathcal{L}_{\text {flow\_ph}}$ and $\mathcal{L}_{\text{art\_mask}}$ for self-supervised training. The supervised final loss function is
\begin{equation}
    \mathcal{L}_{\text{self-sup}}=\lambda_0\mathcal{L}_{\text{geo\_ph}}+\lambda_1\mathcal{L}_{\text{flow\_ph}}+\lambda_2\mathcal{L}_{\text{art\_mask}}.
\end{equation}
where $\lambda_0=100$, $\lambda_1=5$, and $\lambda_2=0.05$. Since the network has several update iterations, we use $\gamma=0.9$ to apply the loss to the output of each iteration with exponentially increasing weights. For semi-supervised learning, we modify the $\mathcal{L}_{\text{art\_mask}}$ to the $\mathcal{L}_{\text{gt\_mask}}$.

\vspace{-1em}
\subsection{Implementation Details}
\subsubsection{Training Details}
DeFlowSLAM is implemented in PyTorch, and we use the LieTorch extension~\cite{teed2021tangent} to perform back-propagation in the tangent space of all groups elements. In the ablation study, we trained DeFlowSLAM on the VKITTI2 dataset~\cite{cabon2020virtual} with 2 RTX-3090 GPUs for 80,000 steps, which took about 2 days. While we trained our network on TartanAir dataset ~\cite{tartanair2020iros} for 250k steps with a batch size of 4 and resolution of $344\times464$. Training on 4 RTX-3090 GPUs takes 5 days. In the monocular setup, we fix the first two poses as the true pose for each training sequence as DROID-SLAM\cite{teed2021droid}, for the network can only recover the trajectory of the camera to the similarity transformation.
In addition, there are still normative degrees of uncertainty during training, which may have an impact on the stability in terms of regulation and gradients.
During training, we set each training example consisting of a 6-frame video sequence. We keep the same constructing strategy in \cite{teed2021droid}, making sure that the average distance between adjacent sequence frames is between 8 pixels and 96 pixels.

\subsubsection{SLAM System Details}
In the inference process, we integrate our networks into a full SLAM system. The SLAM system takes the video stream as input and performs simultaneous localization and mapping. Our system contains a frontend thread and a backend thread, both running at the same time. The mission of the frontend thread is to receive new frames, select keyframes, extract features, and perform local bundle adjustment. While the backend thread performs global bundle adjustment for the entire history of keyframes.
\begin{table}[t]
    \caption{\textbf{Ablation Study of DeFlowSLAM Trained and Tested on Vitual KITTI2 (VK) Dataset.} SS means self-supervised, SM means semi-supervised, SF means single flow, DF means dual flow, and MA means Mask-Agg.

    }
    \resizebox{\linewidth}{!}{%
        \begin{tabular}{l|c c c c c}
            Monocular & VK01 & VK02 & VK06 & VK18 & VK20 \\
            \toprule
            DROID-SLAM*~\cite{teed2021droid} & \underline{1.091} & \textbf{0.025} & 0.113 & 1.156 & 8.285 \\
            \hline
            Ours (SM, DF) & \textbf{0.761} & \underline{0.069} & \underline{0.11} & \underline{0.737} & \textbf{2.546}\\
            \hline

            Ours (SS, SF) & 4.278 & 0.187 & 0.142 & 1.248 & 10.487 \\
            Ours (SS, DF) & 1.099 & 0.094 & 1.984 & 0.983 & 6.918 \\
            \textbf{Ours (SS, DF, MA)} & 1.341 & 0.089 & \textbf{0.092} & \textbf{0.376} & \underline{5.975} \\

            \bottomrule
        \end{tabular}
        \label{table:vkitti ablation study}
    }
    \vspace{-1em}
\end{table}

\begin{table}[t]
    \caption{\textbf{Ablation Study of Dynamic Threshold $\mu$ of DeFlowSLAM Trained and Tested on Vitual KITTI2 (VK) Dataset in Self-Supervised Setting.} $\mu=0.5$ achieves the best results.

    }
    \resizebox{\linewidth}{!}{%
        \begin{tabular}{l|c c c c c}
            $\mu$ & VK01 & VK02 & VK06 & VK18 & VK20 \\
            \toprule
            0.3 & 0.763 & 0.117 & 0.126 & 36.10 & 114.5 \\
            0.5 & 1.341 & \textbf{0.089} & \textbf{0.092} & \textbf{0.376} & \textbf{5.975} \\
            0.7 & \textbf{0.738} & 0.274 & 0.324 & 27.41 & 104.1 \\

            \bottomrule
        \end{tabular}
        \label{table:dynamic threshold}
    }
    \vspace{-1em}
\end{table}

\begin{figure}[t]
   \centering
    \includegraphics[width=\linewidth]{pictures/weight.pdf}
    \vspace{-2em}
    \caption{\textbf{DeFlowSLAM Confidence.} We visualize the confidence of our DeFlowSLAM vs. DROID-SLAM. Our method can exploit the dynamic pixels to solve the camera pose. The darker color means the greater weight. DeFlowSLAM also takes  advantage of more information about the parked vehicles on the roadside.}
    \label{fig:weight}
    \vspace{-1em}
\end{figure}
\begin{table}[t]
    \centering
    \caption{\textbf{Dynamic SLAM Results on KITTI (K) \& Virtual KITTI2 (VK) Datasets with Metric: ATE[m] Trained on TartanAir Dataset.} We achieve the best results.}

    \resizebox{\linewidth}{!}
    {%
    \begin{tabular}{l c c c c c c c}
        \toprule
        Method & K09 & K10 & VK01 & VK02 & VK06 & VK18 & VK20 \\
        \midrule
        DSO~\cite{engel2017direct} & \underline{28.1} & 24.0 & - & - & - & - & - \\
        DynaSLAM*~\cite{bescos2018dynaslam} & 41.91 & \underline{7.519} & 27.830 & X & X & X & \underline{2.807} \\
        DROID-SLAM*~\cite{teed2021droid} & 47.1 & 11.0 & \underline{2.259} & \underline{0.049} & \underline{0.136} & \underline{1.170} & 6.998 \\
        \midrule
        Ours & \textbf{27.8} & \textbf{4.2} & \textbf{0.591} & \textbf{0.021} & \textbf{0.13} & \textbf{0.400} & \textbf{1.039} \\
        \bottomrule
    \end{tabular}
    }
    \label{tab:Kitti}
    \vspace{-1em}
\end{table}

\begin{figure}[t]
\centerline{
\begin{tabular}{cc}
\multicolumn{2}{c}{\includegraphics[width=0.9\linewidth]{pictures/vkitti01.jpg}} \\
\includegraphics[width=0.47\linewidth]{pictures/vkitti02.png} &
\includegraphics[width=0.47\linewidth]{pictures/vkitti20.png} \\
\end{tabular}
}
    \caption{\textbf{Comparison Trajectory Results of Our Method with DROID-SLAM on VKITTI2 Sequences 01 (Top), 02 (Bottom-Left), and 20 (Bottom-right).} In these dynamic sequences, our method performs better than DROID-SLAM, having better trajectory estimation results.}
  \label{fig: trajetory of vkitti2}
  \vspace{-1em}
\end{figure}

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{pictures/kitti0910.png}
    \label{fig:sub2}
     \vspace{-2em}
    \caption{\textbf{Trajectory Comparison between Our Method and DROID-SLAM.} In KITTI sequences 09 (Left) and 10 (Right), our trajectories are closer to the ground truth.}
    \vspace{-1em}
\end{figure}

\noindent\textbf{Initialization.}
Following DROID-SLAM~\cite{teed2021droid}, our system keeps collecting frames until having 12 frames in total. The system saves the old frame only if its estimated static flow is greater than 16 pixels. Once the system has gotten 12 frames, it initializes the factor graph by creating edges between keyframes within 3 timesteps. Then the system will run the dynamic update operator 10 times using edges in this graph.

\noindent\textbf{Frontend.}
The frontend directly processes the incoming video stream, which maintains a set of keyframes and a frame graph storing edges between co-visible keyframes, where the keyframe poses and depths are being actively optimized. When a new frame comes, we first extract its feature map, then add it to the frame graph, building edges with its 3 nearest neighbors. The distance used here is measured by averaging static flow. The new frame's pose is initialized by a linear motion model, and several update iterations are performed to optimize the keyframe poses and depths. We fix the first two poses to eliminate the scale uncertainty while treating all depths as free variables. After tracking the new frame, we use the static flow-based distance to remove redundant frames and remove the oldest keyframe if the buffer is overloaded.

\noindent\textbf{Backend.}
The backend performs global bundle adjustment over the entire history of keyframes~\cite{teed2021droid}.
During each iteration, we first choose the edge whose containing frames are temporally adjacent. We then choose the new edge according to the distance map, which is made by exploiting the averaging static flow between all keyframe pairs. Similar to DROID-SLAM~\cite{teed2021droid}, for each chosen edge, we suppress the adjacent edges within a distance of 2, where the distance here is defined as the Chebyshev distance between index pairs $||(i,j) - (k,l)||_\infty = \max(|i-k|, |j-l|)$. We then apply the dynamic update operator to the new complete frame graph.

\noindent\textbf{Stereo and RGB-D Inputs.}
\label{sec:stereo and rgbd}
Our system can also use stereo and RGB-D inputs. For RGB-D, we still treat depth as a variable to optimize, since the sensor depth can be noisy. We simply add a term to the optimization process which is shown in Eq.~(\ref{eqn:objective}). It will then penalize the squared distance between the predicted depths and the sensor depths. For stereo, we use the same system, except for doubling the frames and fixing the relative pose between the left and right frames in the DBA layer.
We can exploit the stereo evidence by adding crossed camera edges in the optimization graph~\cite{teed2021droid}.

\vspace{-1em}

\section{Experiments}

We first validate the effectiveness of our method in the highly dynamic scenarios of Virtual KITTI2~\cite{cabon2020virtual} in the ablation study. Further, we train DeFlowSLAM from scratch with the same strategy on a larger dataset, TartanAir~\cite{tartanair2020iros}, and test the generalization of our method on different dynamic datasets, such as Virtual KITTI2~\cite{cabon2020virtual}, KITTI~\cite{geiger2013vision}, and dynamic sequences of TUM-RGBD\cite{schubert2018tum}. We also test monocular or stereo datasets, such as static scenes of TUM-RGBD~\cite{schubert2018tum} and EuRoc~\cite{burri2016euroc}.
Following previous works~\cite{teed2021droid}, we use absolute trajectory error (ATE)~\cite{tumrgbd} to evaluate the accuracy of the estimated camera trajectories. In particular, we compare DeFlowSLAM to DynaSLAM~\cite{bescos2018dynaslam}, etc in dynamic scenes, demonstrating the effectiveness of our method. We also compare DeFlowSLAM to DROID-SLAM in AR applications, the results further prove the robustness of our pose estimation.

\begin{table*}[t]
	\centering
	\caption{\textbf{Dynamic SLAM Results on TUM Dynamic Sequences with Metric: ATE[m].} The best results are shown in bold. DeFlowSLAM achieves competitive and even best performance. Note that DVO SLAM, ORB-SLAM2 and PointCorr use the RGB-D dataset, while our method and DROID-SLAM only use the monocular RGB dataset.
	}
	\label{tab:tum_dy}
	\resizebox{14cm}{!}
	{
	\begin{tabular}{cc|ccc|cc}
		\toprule
		\multicolumn{2}{c|}{\multirow{1}{*}{Sequences}}         & \multicolumn{5}{c}{ Trans. RMSE of  trajectory alignment [m]}    \\
		\multicolumn{2}{c|}{}       &  DVO SLAM~\cite{kerl2013robust}
    &
     ORB-SLAM2~\cite{orbslam2}
		&
     PointCorr~\cite{dai2020rgb} & DROID-SLAM~\cite{teed2021droid} & Ours
     \\
		\midrule
		\multirow{5}{*}{slightly dynamic}  & fr2/desk-person   &0.104 &\textbf{0.006} &0.008  & 0.017          & 0.013  \\
		& fr3/sitting-static     &0.012      & 0.008  & 0.010  & \textbf{0.007} &\textbf{0.007}   \\
		& fr3/sitting-xyz        &0.242      & 0.010  & \textbf{0.009}   &0.016  &0.015  \\
		& fr3/sitting-rpy       &0.176    & 0.025    &\textbf{0.023}   & 0.029  & 0.027   \\
		& fr3/sitting-halfsphere  &0.220   & 0.025  & 0.024  & \textbf{0.022}  & 0.025  \\
		\midrule
		\multirow{4}{*}{highly dynamic} & fr3/walking-static   &0.752 & 0.408 & 0.011 & 0.016 & \textbf{0.007}  \\
		& fr3/walking-xyz      &1.383    &0.722  & 0.087  & 0.019 & \textbf{0.018}  \\
		& fr3/walking-rpy     &1.292     &0.805 & 0.161  &0.059 &  \textbf{0.057}    \\
		& fr3/walking-halfsphere  &1.014   & 0.723   & \textbf{0.035} & 0.312  &   0.42   \\
		\bottomrule
	\end{tabular}
	}
	 \vspace{-1em}
\end{table*}

\begin{table*}[h]
    \centering
    \caption{
        \textbf{Monocular SLAM Results on TartanAir Monocular Benchmark with Metric: ATE[m]}. Bold stands for best results and underlined for the second best. * means the results are generated by running the official pretrained model in our environment to ensure evaluation consistency. X means the system has failed here. - means lack of result. We achieve the best results.
    }
    \resizebox{.8\linewidth}{!}{%
        \begin{tabular}{l|cccccccc | c}
            Monocular & MH000 & MH001 & MH002 & MH003 & MH004 & MH005 & MH006 & MH007 & Avg \\
            \toprule
            ORB-SLAM~\cite{mur2015orb} & 1.30 & \textbf{0.04} & 2.37 & 2.45 & X & X & 21.47 & 2.73 & - \\
            DeepV2D~\cite{deepv2d} & 6.15 & 2.12 & 4.54 & 3.89 & \underline{2.71} & 11.55 & 5.53 & 3.76 & 5.03 \\
            TartanVO~\cite{tartanvo2020corl} & 4.88	& 0.26 & 2.00 & 0.94 & \textbf{1.07} & 3.19 & 1.00 & 2.04 & 1.92 \\
            DROID-SLAM*~\cite{teed2021droid} & \textbf{0.04} & 0.69 & \underline{0.03} & \underline{0.02} & 3.73 & \underline{0.62} & \underline{0.38} & \textbf{0.07} & \underline{0.70} \\
            Ours & \underline{0.63} & \underline{0.06} & \textbf{0.02} & \textbf{0.01} & 2.80 & \textbf{0.20} & \textbf{0.31} & \underline{0.45} & \textbf{0.56} \\
            \midrule
        \end{tabular}
        \label{table:TartanAir mono}
    }
    \vspace{-1em}
\end{table*}

\begin{table*} [h]
    \caption{
        \textbf{Monocular SLAM Results on EuRoC Dataset with Metric: ATE[m]}. $^\dagger$ denotes visual odometry methods. X means the system has failed here. - means lack of result. We achieve comparable results on par with DROID-SLAM.
    }
    \centering
    \resizebox{.9\linewidth}{!}{%
        \begin{tabular}{cl| ccccc | ccc | ccc | c}
            & & MH01 & MH02 & MH03 & MH04 & MH05 & V101 & V102 & V103 & V201 & V202 & V203 & Avg \\
            \toprule
            \multirow{4}{*}{\rotatebox[origin=c]{90}{Deep/Hyb.}}
            & DeepFactors~\cite{deepfactors} & 1.587 & 1.479 & 3.139 & 5.331 & 4.002 & 1.520 & 0.679 & 0.900 & 0.876 & 1.905 & 1.021 & 2.040 \\
            & DeepV2D~\cite{deepv2d}$^\dagger$ & 0.739 & 1.144 & 0.752 & 1.492 & 1.567 & 0.981 & 0.801 & 1.570 & 0.290 & 2.202 & 2.743 & 1.298 \\
            & DeepV2D (Tartan Air)$^\dagger$ & 1.614 & 1.492 & 1.635 & 1.775 & 1.013 & 0.717 & 0.695 & 1.483 & 0.839 & 1.052 & 0.591 & 1.173 \\
            & TartanVO~\cite{tartanvo2020corl}$^\dagger$ & 0.639 & 0.325 & 0.550 & 1.153 & 1.021 & 0.447 & 0.389 & 0.622 & 0.433 & 0.749 & 1.152 & 0.680 \\
            & D3VO + DSO~\cite{d3vo}$^\dagger$ & - & - & 0.08 \ \ & - & 0.09 \ \ & - & - & 0.11 \ \ & - & 0.05 \ \ & \underline{0.19} \ \ & - \\
            \midrule
            \multirow{5}{*}{\rotatebox[origin=c]{90}{Classical}}
            & ORB-SLAM~\cite{mur2015orb} & 0.071 & 0.067 & 0.071 & 0.082 & 0.060 & \textbf{0.015} & 0.020 & X & 0.021 & \underline{0.018} & X & - \\
            & DSO~\cite{dso}$^\dagger$ & 0.046 & 0.046 & 0.172 & 3.810 & 0.110 & 0.089 & 0.107 & 0.903 & 0.044 & 0.132 & 1.152 & 0.601 \\
            & SVO~\cite{svo}$^\dagger$ & 0.100 & 0.120 & 0.410 & 0.430 & 0.300 & 0.070 & 0.210 & X & 0.110 & 0.110 & 1.080 & - \\
            & DSM~\cite{dsm} & 0.039 & 0.036 & 0.055 & \underline{0.057} & 0.067 & 0.095 & 0.059 & 0.076 & 0.056 & 0.057 & 0.784 & \underline{0.126} \\
            & ORB-SLAM3~\cite{orbslam3} & \underline{0.016} & \underline{0.027} & 0.028 & 0.138 & 0.072 & 0.033 & 0.015 & \underline{0.033} & 0.023 & 0.029 & X & - \\
            \midrule
            & DROID-SLAM*~\cite{teed2021droid} & \textbf{0.013} & \textbf{0.014} & \underline{0.022} & \textbf{0.043} & \textbf{0.043} & 0.037 & \underline{0.012} & \textbf{0.020} & \underline{0.017} & \textbf{0.013} & \textbf{0.014} & \textbf{0.022} \\
            & Ours & 0.018 & 0.037 & \textbf{0.020} & 0.060 & \underline{0.048} & \underline{0.031} & \textbf{0.008} & 0.083 & \textbf{0.016} & 0.035 & 1.137 & 0.136 \\
            \midrule
        \end{tabular}
    }
    \label{table:EurocMono}
     \vspace{-1em}
\end{table*}

\begin{table*}[h]
    \caption{
        \textbf{Monocular SLAM Results on TUM-RGBD Dataset with Metric: ATE[m]}. X means the system has failed here. - means lack of result. We achieve comparable results on par with DROID-SLAM.
    }
    \centering
    \resizebox{.75\linewidth}{!}{%
        \begin{tabular}{l|ccccccccc | c}
            & 360 & desk & desk2 & floor & plant & room & rpy & teddy & xyz & avg \\
            \toprule
            ORB-SLAM2~\cite{orbslam2} & X & 0.071 & X & \underline{0.023} & X & X & X & X & 0.010 & - \\
            ORB-SLAM3~\cite{orbslam3} & X & \underline{0.017} & 0.210 & X & \underline{0.034} & X & X & X & \textbf{0.009} & - \\
            \midrule
            DeepTAM~\cite{deeptam} & \textbf{0.111} & 0.053 & 0.103 & 0.206 & 0.064 & \underline{0.239} & 0.093 & 0.144 & 0.036 & 0.116 \\
            TartanVO~\cite{tartanvo2020corl} & 0.178 & 0.125 & 0.122 & 0.349 & 0.297 & 0.333 & 0.049 & 0.339 & 0.062 & 0.206\\
            DeepV2D~\cite{deepv2d} & 0.243 & 0.166 & 0.379 & 1.653 & 0.203 & 0.246 & 0.105 & 0.316 & 0.064 & 0.375 \\
            DeepV2D (TartanAir) & 0.182 & 0.652 & 0.633 & 0.579 & 0.582 & 0.776 & 0.053 & 0.602 & 0.150 & 0.468 \\
            DeepFactors~\cite{deepfactors} & 0.159 & 0.170 & 0.253 & 0.169 & 0.305 & 0.364 & 0.043 & 0.601 & 0.035 & 0.233\\
            \midrule
            DROID-SLAM*~\cite{teed2021droid} & \textbf{0.111} & 0.018 & \underline{0.042} & \textbf{0.021} &
            \textbf{0.016} & \textbf{0.049} & \underline{0.026} & \underline{0.048} & 0.012 & \textbf{0.038} \\
            Ours  & 0.159 & \textbf{0.016} & \textbf{0.030} & 0.169 &
            0.048 & 0.538 & \textbf{0.021} & \textbf{0.039} & \textbf{0.009} & \underline{0.114} \\
            \midrule
        \end{tabular}
    }
    \label{table:TUMmono}
     \vspace{-1em}
\end{table*}

\begin{table*}[h]
    \centering
    \caption{
        \textbf{Stereo SLAM results on TartanAir Stereo Benchmark with Metric: ATE[m]}. X means the system has failed here. - means lack of result. We achieve comparable results on par with DROID-SLAM.
    }
    \resizebox{.8\linewidth}{!}{%
        \begin{tabular}{l|cccccccc | c}
            Stereo & SH000 & SH001 & SH002 & SH003 & SH004 & SH005 & SH006 & SH007 & Avg \\
            \toprule
            ORB-SLAM2~\cite{orbslam2} & \textbf{0.05} & 6.67 & X & X & X & X & 0.10 & X & -\\
            TartanVO~\cite{tartanvo2020corl} & 2.52 & 1.61 & 3.65 & 0.29 & 3.36 & \underline{4.74} & 3.72 & 3.06 & 2.87 \\
            DROID-SLAM*~\cite{teed2021droid} & 0.44 & \textbf{0.08} & \textbf{0.13} & \underline{0.20}  & \underline{0.16} & \textbf{3.29} & \underline{0.38} & \underline{0.18} & \textbf{0.61} \\
            Ours & \underline{0.14} & \underline{0.10} & \textbf{0.13} & \textbf{0.08}
             & \textbf{0.09} & 7.60 & \textbf{0.03} & \textbf{0.02} & \underline{1.02} \\
            \midrule
        \end{tabular}
    }
    \label{table:TartanAir stereo}
     \vspace{-1em}
\end{table*}

\begin{table*} [h]
    \caption{
        \textbf{Stereo SLAM results on EuRoC Dataset with Metric: ATE[m]}. - means lack of result. We achieve comparable results on par with DROID-SLAM.
    }
    \centering
    \resizebox{.9\linewidth}{!}{%
        \begin{tabular}{l| ccccc | ccc | ccc | c}
            & MH01 & MH02 & MH03 & MH04 & MH05 & V101 & V102 & V103 & V201 & V202 & V203 & Avg \\
            \toprule
            D3VO + DSO~\cite{d3vo} & - & - & 0.08 \ \ & - & 0.09 \ \ & - & - & 0.11 \ \ & - & 0.05 \ \ & - & - \\
            ORB-SLAM2~\cite{orbslam2} & 0.035 & 0.018 & 0.028 & 0.119 & 0.060 & \underline{0.035} & 0.020 & \underline{0.048} & \underline{0.037} & 0.035 & - & - \\
            VINS-Fusion~\cite{vinsf} & 0.540 & 0.460 & 0.330 & 0.780 & 0.500 & 0.550 & 0.230 & - & 0.230 & 0.200 & - & - \\
            SVO~\cite{svo} & 0.040 & 0.070 & 0.270 & 0.170 & 0.120 & 0.040 & 0.040 & 0.070 & 0.050 & 0.090 & 0.790 & 0.159 \\
            ORB-SLAM3~\cite{orbslam3} & 0.029 & 0.019 & \underline{0.024} & 0.085 & 0.052 & \underline{0.035} & 0.025 & 0.061 & 0.041 & \underline{0.028} & 0.521 & 0.084 \\
            \midrule
            DROID-SLAM*~\cite{teed2021droid} & \textbf{0.015} & \underline{0.013} & 0.035 & \textbf{0.048} & \textbf{0.040} & 0.037 & \textbf{0.011} & \textbf{0.020} & \textbf{0.018} & \textbf{0.015} & \textbf{0.017} & \textbf{0.024} \\
            Ours & \underline{0.018} & \textbf{0.011} & \textbf{0.020} & \underline{0.061} & \underline{0.042} & \textbf{0.031} & \textbf{0.011} & 0.090 & 0.039 & 0.037 & \underline{0.478} & \underline{0.075} \\
            \midrule
        \end{tabular}
    }
    \label{table:EurocStereo}
     \vspace{-1em}
\end{table*}
\vspace{-1em}
\subsection{Datasets}
\noindent\textbf{Virtual KITTI2}~\cite{cabon2020virtual} consists of 5 sequence clones from the KITTI tracking benchmark, and provides different variants of these sequences such as modified weather conditions (e.g. fog, rain) or modified camera configurations (e.g. rotated by $15^{\circ}$).
In the ablation study, we use the split \textit{clone} as the training set, the split \textit{15-degree rotation} as the validation set, and the split \textit{30-degree rotation} of as the test set.

\noindent\textbf{TartanAir}~\cite{tartanair2020iros} is a challenging dataset for robot navigation tasks and SLAM. The data is collected in photo-realistic simulation environments with the presence of moving objects, changing light, and various weather conditions. We use the official test split ~\cite{tartanair2020iros} and calculate ATE in all sequences.

\noindent\textbf{KITTI}~\cite{geiger2013vision} is a dataset capturing real-world traffic situations and ranges from freeways over rural areas to inner-city scenes with many static and dynamic objects. It contains object labels in the form of 3D tracklets that provides online benchmarks for stereo, optical flow, object detection, and visual odometry.

\noindent\textbf{TUM-RGBD}~\cite{schubert2018tum} consists of indoor scenes captured with a handheld camera. This is a notoriously difficult dataset for monocular methods due to rolling shutter artifacts, motion blur, and heavy rotation~\cite{teed2021droid}. We use the dynamic sequences to evaluate the effectiveness of DeFlowSLAM in dynamic scenes and exploit the static sequences in monocular or stereo settings.

\noindent\textbf{EuRoc}~\cite{burri2016euroc} is a dataset containing 11 sequences classified as simple, medium and difficult based on illumination and camera motion. The dataset \cite{d3vo} is very challenging due to the strong motion and the significant illumination changes between stereo and temporal images. It is a widely used benchmark to evaluate SLAM systems.

\vspace{-1em}
\subsection{Ablation Study}
We conduct an ablation study to verify the effectiveness of our dual-flow representation in dynamic scenes and further explore the self-supervised training to train our model. We chose Virtual KITTI2 dataset \cite{cabon2020virtual} for ablation study.

\noindent\textbf{Framework Design.} To remove the interference of dynamic objects, an easy way to implement this is to add a module to DROID-SLAM that removes dynamic objects, which is considered a mask-removal approach. Instead, we use our dual-flow representation to keep more pixels for solving the pose, decomposing the static components even for regions of dynamic objects. Fig.~\ref{fig:weight} shows that DeFlowSLAM can exploit the dynamic pixels to solve the camera pose, compared with DROID-SLAM~\cite{teed2021droid}. Tab.~\ref{table:vkitti ablation study} shows that the dual-flow representation is better than the rough single-flow method. Notice that SS means self-supervised, SM means semi-supervised, SF means single flow, and DF means dual flow.

\noindent\textbf{Supervision Policy.} We explore the possibility of training DeFlowSLAM in a self-supervised manner. Compared with the supervised approach, which requires a large number of complicated loss functions and is difficult to adjust the appropriate loss weights, and the semi-supervised approach, which requires additional masks for dynamic objects, the self-supervised approach achieves the transformation of different loss functions, allowing a less restricted adaptation to new scenarios and demonstrating better generalization ability. Tab. \ref{table:vkitti ablation study} demonstrates that the self-supervised approach can achieve comparable or better accuracy than the supervised approach.

\noindent\textbf{Mask-Agg Setting.}
In self-supervised training, we introduce the Mask-Agg technique to reduce the influence of error matching of the pixels. Because the predicted dynamic masks in self-supervised training are not converged in the early stage of training, we only introduce Mask-Agg at the last $1/10$ epochs. Tab. \ref{table:vkitti ablation study} demonstrates that Mask-Agg helps to improve pose estimation, where MA means Mask-Agg.

\noindent\textbf{Dynamic Threshold $\mu$.}
In self-supervised setting, we perform an ablation study of dynamic threshold $\mu$ on Virtual KITTI2 (VK) dataset. Experiments demonstrate that $\mu=0.5$ achieves the better performance, as shown in Tab.~\ref{table:dynamic threshold}.

\vspace{-1em}
\subsection{Generalization} After verifying the validity of our hypothesis, we trained our DeFlowSLAM on TartanAir dataset~\cite{tartanair2020iros} from scratch and test our DeFlowSLAM on other popular SLAM datasets, such as Virtual KITTI2 (VKITTI2) \cite{cabon2020virtual} and KITTI \cite{geiger2013vision} for autonomous driving scenarios with dynamic objects, EuRoc \cite{burri2016euroc} for drones with strong motion and significant illumination changes, and TUM RGB-D \cite{schubert2018tum} for hand-held SLAM with motion blur and heavy rotations. The results demonstrate that our model generalizes well to different datasets, shown in Figure \ref{fig:slam reconstruction}. Next, we show the effectiveness of DeFlowSLAM in several settings.

\begin{figure*}[t]
  \centering
  \includegraphics[width=\linewidth]{pictures/AR_gt_deflow_droid_original_18.png}
  \vspace{-2em}
  \caption{\textbf{AR Application.} DeFlowSLAM deals with the dynamic objects in the scene very well while DROID-SLAM exhibits significant drifts~(the red boxes). We augment the original video with a virtual tree, a car, and a street lamp. From left to right and from top to down: Ground-Truth, DeFlowSLAM, DROID-SLAM, Original Image. }
  \label{fig:AR demo}
  \vspace{-1em}
\end{figure*}
\vspace{-1em}

\begin{table}[t]
    \caption{\textbf{Motion Segmentation.} We show the motion segmentation results of DeFlowSLAM on Virtual KITTI2 dataset.

    }
    \resizebox{\linewidth}{!}{%
        \begin{tabular}{l|c c c c c}
            Monocular & VK01 & VK02 & VK06 & VK18 & VK20 \\
            \toprule
            Ours  &0.538585	&0.528356	&0.562788	&0.739282	&0.654025\\
            \bottomrule
        \end{tabular}
        \label{table:vkitti motion segmentation}
    }
    \vspace{-1em}
\end{table}

\begin{figure}[t]
\label{fig:motion segmentation}
  \centering
  \includegraphics[width=\linewidth]{pictures/motion_segmentation_gt_pt.png}
    \vspace{-2em}
  \caption{\textbf{Motion Segmentation.} From left to right: Ground-Truth dynamic mask, Predicted dynamic mask larger than the dynamic threshold $\mu=0.5$.}
  \vspace{-1em}
\end{figure}

\subsection{Dynamic SLAM}
We test the performance of the proposed DeFlowSLAM on sequences 09 and 10 from KITTI dataset \cite{geiger2013vision} and all sequences from Virtual KITTI2 dataset \cite{cabon2020virtual}. Here we provide an additional trajectory for this experiment. The ATE results are shown in Table \ref{tab:Kitti}. Compared with DynaSLAM \cite{bescos2018dynaslam} which uses Mask-RCNN for dynamic environment and DROID-SLAM \cite{teed2021droid}, our DeFlowSLAM is far more accurate and robust in dynamic scenes. We also perform evaluations on TUM RGB-D dynamic sequences with different dynamic proportions and the comparison results in Tab.~\ref{tab:tum_dy} shows that DeFlowSLAM achieves competitive and even best performance. Note that DVO SLAM~\cite{kerl2013robust} , ORB-SLAM2~\cite{orbslam2} and PointCorr~\cite{dai2020rgb} use the RGB-D dataset, while our method and DROID-SLAM~\cite{teed2021droid} only use the monocular RGB dataset.

\vspace{-1em}
\subsection{Monocular SLAM}
In monocular setting, we test our trained DeFlowSLAM on TartanAir test sets, EuRoC, and TUM RGB-D dataset.
As shown in Tab.~\ref{table:TartanAir mono}, DeFlowSLAM achieves the best results.
Tab.~\ref{table:EurocMono} and Tab.~\ref{table:TUMmono} show that our method achieves comparable even better results than the SOTA supervised method, DROID-SLAM~\cite{teed2021droid} in most sequences. The results also demonstrate our DeFlowSLAM is more robust than the classical SLAM algorithms as they failed in many sequences. Specifically, we achieve an average ATE of 0.136m on EuRoC dataset in the monocular setting, and an average ATE of 0.114m on TUM-RGBD static sequences, outperforming most supervised methods.

\vspace{-1em}
\subsection{Stereo SLAM}
Under stereo setup, our trained DeFlowSLAM is also tested on TartanAir test dataset and EuRoC stereo dataset. Tab.~\ref{table:TartanAir stereo} illustrates  DeFlowSLAM achieves comparable results on par with DROID-SLAM~\cite{teed2021droid}, with an average ATE of 1.02m on TartanAir stereo test dataset, outperforming TartanVO~\cite{tartanvo}. Tab.~\ref{table:EurocStereo} shows that DeFlowSLAM exhibits comparable results on EuRoc dataset in the stereo setting with DROID-SLAM~\cite{teed2021droid}, outperforming most supervised methods and traditional SLAM, ORB-SLAM3~\cite{orbslam3}. In most sequences, our method is on the same order of magnitude as DROID-SLAM~\cite{teed2021droid}, which shows the effectiveness of our method.

\vspace{-1em}
\subsection{AR Applications}
We conduct extensive experiments on AR applications to demonstrate the robustness of DeFlowSLAM. As shown in Fig. \ref{fig:AR demo}, we augment the original video with a virtual tree, a car, and a street lamp. Our DeFlowSLAM can deal with the dynamic objects in the scene very well while DROID-SLAM exhibits significant drifts (the red boxes).

\vspace{-1em}
\subsection{Motion Segmentation}
Although our method focuses on the SLAM system, our dual-flow representation can be well applied to motion segmentation. We simply set a threshold value (such as $\mu=0.5$) for motion and visualize the pixel points of the dynamic field larger than this threshold to obtain the result of motion segmentation, as shown in Fig.~\ref{fig:motion segmentation} and Tab.~\ref{table:vkitti motion segmentation}.

\vspace{-1em}
\section{Discussion and Limitations}
Although our system is versatile to many SLAM settings and is more robust to the challenges of dynamic scenes, it can be further improved in the following directions. DeFlowSLAM performs slightly weaker in some scenarios than DROID-SLAM, probably because we use a certain fixed dynamic threshold. We can explore dynamic threshold estimation methods to the challenges of different scenarios.
As with droid-SLAM, we have high memory requirements for longer sequences and larger scenes, and our DeFlowSLAM system needs to run in segments. A lightweight and efficient SLAM system is a potential research direction.
For some loop closure sequences, we can add loop closure constraints to reduce the drift.
Our system focuses more on solving the camera pose, and the depth and optical flow obtained are only 1/8 of the original image size, which is not ideal for tasks like depth estimation and optical flow estimation. An efficient and versatile SLAM system is worthwhile researching. We can also explore dynamic object reconstruction with dynamic dense SLAM.
\vspace{-1em}
\section{Conclusions}
We present a novel dual-flow representation that decomposes the optical flow into a static flow field caused by the camera poses and a dynamic field caused by the dynamic objects' motion. We present dynamic update module with dual-flow representation to compose the full SLAM system, DeFlowSLAM.
We explore a self-supervised method to train our dual-flow based dynamic dense SLAM system, which outperforms DROID-SLAM in highly dynamic scenes and achieves comparable performance in static and slightly dynamic scenes.

\vspace{-1em}

\section*{Acknowledgments}
This work was partially supported by NSF of China (No. 61932003) and ZJU-SenseTime Joint Lab of 3D Vision.

\bibliographystyle{IEEEtran}


\section{Introduction}
Adversarial attacks \cite{Szegedy2014, Goodfellow2015, Papernot2016limitations, Moosavi2016, CW2017, Madry2018, Athalye2018obfuscated} add small, imperceptible but well optimized noise to the clean image which can elicit an incorrect decision from the model.
These attack methods  \cite{poursaeed2018generative, dong2017discovering, xie2019improving, inkawhich2020perturbing, gao2020patch, wang2021admix} craft adversarial examples that can broadly be categorized based on how much information is available about the target model. In a \emph{white-box} attack setting, the attacker has complete knowledge of the target model and can directly optimize adversarial perturbations for the given model. In a more realistic \emph{black-box} attack setting, the attacker does not have access to the target model, its architectural details or targeted task (\emph{e.g}\bmvaOneDot, classification, segmentation or object detection). In such a case, adversarial examples are created on a surrogate model and then transferred to the black-box model.  Adversarial examples generated from surrogate models trained on a large-scale dataset in a supervised manner have better transferability \cite{naseer2019cross, dong2018boosting}. Transferability of such attacks improves further by fine-tuning the surrogate model to enhance their representation capacity \emph{e.g}\bmvaOneDot, by finding better self-ensemble from a given pretrained model \cite{naseer2021improving}. The adversarial transferability hence depends on the generalizability of the surrogate model. Such attacks are also restricted by the availability of a pretrained surrogate and information about the label space.

In this work, we call into question this assumption and consider a stronger threat model where an attack is launched from few unannotated or cross-domain samples (\emph{e.g}\bmvaOneDot, painting to ImageNet samples) without any knowledge of target networks or tasks. This threat model poses a challenge on how to learn an effective surrogate model from the limited unlabelled data and then how to generate self-supervised transferable adversarial examples.

With limited availability of data, neural networks can easily memorize the data \cite{zhang2021understanding} even with strong augmentations \cite{li2020practical}. Therefore, Li \emph{et al}\bmvaOneDot \cite{li2020practical} propose to reconstruct transformed input images to learn a surrogate model. However, their effective surrogate training and attack approach requires supervision though annotated data samples. In order to reduce overfitting over few data samples and to find robust features, we take inspiration from adversarial training \cite{Goodfellow2015, springer2021little} and propose self-supervised Adversarial Pixel Restoration to train a surrogate model. The min-max objective of our proposed training allows to find a flatter minima with robust features which compliments our self-supervised adversarial attack to achieve higher adversarial transferability. Our main contributions are as follows:
\begin{itemize}
    \item We propose self-supervised Adversarial Pixel Restoration to find highly transferable patterns by learning over flatter loss surfaces. Our training approach allows launching cross-domain attacks without access to large-scale labeled data or pretrained models.
    \item Our adversarial attack is self-supervised in nature and  independent of any task-specific objective. For instance, our approach optimizes the robust transformed loss surface of the surrogate via fooling its reconstruction ability. This allows to generate task independent adversaries. Therefore, our approach can transfer perturbations to a variety of tasks as we demonstrate for classification, object detection, and segmentation.
    \item We provide a thorough analysis  of our proposed method to establish it's effectiveness. We observe that our approach leads to smoother loss landscapes (Fig. \ref{Loss surfaces}), helping in crafting more generalizable adversarial examples. Our method remains effective even in extreme data scarcity \emph{e.g}\bmvaOneDot when trained on two data samples only. (see Sec. \ref{subsec: Ablative Analysis}).

\end{itemize}

\section{Related Work}

Several gradient-based methods  \cite{Goodfellow2015, kurakin2018adversarial, mlatscale, ensembleadv} have been proposed for crafting adversarial examples directly on the target classifiers. However, when the access to the target model is limited to just a finite amount of queries, current methods either rely on the transferability of surrogate models \cite{DasTdatafree, Papernot2016transferability} \cite{Papernot2017} or estimate the gradients/boundary of the target model \cite{ZOOgradest, decisonbased_Brendel2018, decison_based_Narodytska2017}. Both of the above approaches require a non-trivial number of queries from the target model, making it highly impractical in real-case scenarios. A practical threat model was introduced in \cite{naseer2019cross}, where the attacker does not have access to the training distribution of the target model, as well as querying is prohibited. Authors train a generator-based surrogate model with the help of training data and a pretrained classifier obtained from a different domain than the target models. However, the pretrained classifier as well as the generator are trained on a large annotated dataset.

In \cite{li2020practical}, a stronger threat model with access to limited data samples (order of tens) was proposed. Inspired from self-supervised learning methods, autoencoder-based surrogate models are trained with limited data. However, the transferability of the autoencoders trained in an unsupervised manner is still moderate. Most of the previous works have explored the transferability of surrogate  models trained on the target model's training set. In  \cite{springer2021adversarial}, the authors hypothesize that the features of robust classifiers are more universal than  non-robust classifiers, which lead to adversarial examples that are more transferable.  Building on this, \cite{springer2021little}  observe that having a classifier adversarially trained with a small perturbation budget, leads to highly transferable adversarial examples.
Unlike prior works, we focus on constructing robust surrogate models in a fully unsupervised manner. We consider  the case of training robust model with limited data
to improve adversarial transferability. Furthermore, we also consider a practical scenario of availability of large unlabelled dataset for training of robust models which can be used for constructing cross-domain and cross-task  adversaries
\section{Adversarial Pixel Restoration for Transferable Perturbations}
Our goal is to learn a surrogate model, $\mathcal{F}$, from a given unlabelled data distribution, $P_s$, with a set of only few image samples ($\leq 20$). This setting is in contrast to the existing transferable adversarial attacks \cite{dong2018boosting, naseer2021improving, xie2019improving, Naseer_2021_ICCV} that assume access to a surrogate model trained on a large-scale annotated data (\emph{e.g}\bmvaOneDot, ImageNet \cite{deng2009imagenet}) in a supervised fashion. In the presence of unlabeled data, however, the surrogate model can be trained by defining a self-supervised task, $\mathcal{T}_s$, such as predicting rotation \cite{gidaris2018unsupervised}, solving jigsaw puzzle \cite{noroozi2016unsupervised} or by matching different views of the same input sample \cite{caron2021emerging}. A major challenge is that deep neural networks can easily memorise the data \cite{zhang2021understanding} and quickly overfit the few available samples even after applying strong data augmentation techniques \cite{li2020practical}. This results in a surrogate model with less generalizable representations and consequently, adversarial attack launched from such a model has weak transferability (Sec.~\ref{subsec: Transferability from Few In-Domain Samples}).

We propose adversarial pixel restoration as a prior to train the surrogate model $\mathcal{F}$ that boosts transferability of the adversarial attacks (see Fig. \ref{fig: concept_fig} ). We create adversarial examples from the original input images by attacking the source model in the pixel space using our proposed adversarial attack (Sec.~\ref{subsec: Adversarial Pixel Transformations}). Our approach also shifts the position of input pixels via transformations such as rotation or jigsaw shuffle. We then train a model to denoise and restore pixels to their original positions via our proposed adversarial training (Algo. \ref{algo: Adversarial Denoising and Restoration}).   The surrogate model $\mathcal{F}$ in our case consists of an autoencoder as explained below.

\begin{figure}[t]
\centering
\includegraphics[width=\textwidth]{images/CopyofConceptdrawiodrawio.png} \vspace{-0.7em}
\vspace{-2em}
\caption{\small Our approach trains an autoencoder based surrogate model via self-supervised adversarial pixel restoration to learn generalizable representations from a limited number of data samples ($\le 20$). Our training is based on a min-max strategy. We first generate adversarial examples by fooling model's reconstructive ability \emph{(maximization)}, followed by updating the model parameters based on restoration of the transformed adversarial and clean sample \emph{(minimization)}. Our approach allows launching transferable self-supervised adversarial attacks without any knowledge of target (black-box) model.
}\vspace{-1em}
\label{fig: concept_fig}
\end{figure}

\subsection{Surrogate Architecture}
\label{subsec: Surrogate Architecture}
 The surrogate model is based on an autoencoder \cite{li2020practical}. The encoder consists of a stack of convolution layers. At the beginning of the architecture, convolution layers with a larger kernel size and stride help reduce the spatial resolution of the feature maps, followed by multiple residual blocks where the size of the feature maps is kept constant. The decoder is a lightweight model consisting of two transpose convolution layers to upsample the feature maps. The adversarial training of these autoencoders is based on denoising and pixel restoration as explained next in Sec.~\ref{subsec: Adversarial Training via Denoising}.

\subsection{Adversarial Training via Denoising }
\label{subsec: Adversarial Training via Denoising}
\vspace{-0.3em}
The feature space of a slightly robust classifier produces highly transferable adversarial examples \cite{springer2021little}. However, adversarial training of such models is computationally demanding due to iterative training and also requires a large-scale labeled dataset that might not be accessible to the attacker in real-world scenarios. We assume a more practical threat model, where the attacker has access to a data distribution $P_s$ with limited number of samples without any annotations. We adversarially learn a surrogate model on the data distribution $P_s$ to transfer adversarial perturbations to the target (black-box) models trained for different tasks (\emph{e.g}\bmvaOneDot, classifications, object detection and segmentation) on possibly different target distribution $Q_t\neq P_s$. In such challenging attack settings, we adversarially train an autoencoder $\mathcal{F}$ via min-max training strategy \cite{Goodfellow2015}. At maximization step, we create adversarial examples by fooling the model $\mathcal{F}$ via adversarial pixel transformations with a single step attack (Algo. \ref{algo: Adversarial Denoising and Restoration}). At minimization step, we denoise and restore the feature and pixel space to achieve generalizable loss surfaces (Fig. \ref{Loss surfaces}) which leads to more transferable attack.

\begin{algorithm}[t]
	\caption{
		Adversarial Denoising and Restoration
	}
	\label{algo: Adversarial Denoising and Restoration}
	\begin{algorithmic}[1]
		\Require Source data distribution $P_s$, pixel transformation $\mathcal{T}_s$, attack step size $\delta$, perceptual budget $\epsilon$, balancing paramter $\lambda$, and maximum training iterations $T$
		\Ensure Randomly initialize surrogate model $\mathcal{F}$
		\For {$t\in [1, 2, \ldots T]$}
    		\State Randomly sample from $P_s$: $\bm{x} \sim P_s$
    		\State Initialize adversary $\bm{x}' \leftarrow$ $\bm{x}$
    		\State Optimize  $\bm{x}'$ using Eq. \ref{eq: attack_loss} or \ref{eq: attack_loss_prototypical}: \algorithmiccomment{Adversarial Pixel Transformation}
    		\begin{equation*}
    		    \bm{x}' \gets \bm{x} + \delta\times\text{sign}\left(\nabla\mathcal{L}_{max}\right)
    		\end{equation*}

    		\State Project adversaries within allowed perceptual budget: $\bm{x}' \gets \text{clip}\left(\bm{x}', \bm{x}-\epsilon, \bm{x}+\epsilon \right)$

    		\State Forward-pass $\bm{x}$ and $\bm{x}'$ through model $\mathcal{F}$ and update its parameters $\theta$ by minimizing the loss given in Eq. \ref{eq: overall_loss}: \algorithmiccomment{Pixel Restoration}
    		\begin{equation*}
    		    \theta \gets \theta - \alpha\times\nabla\mathcal{L}_{min},
    		\end{equation*}
    		where $\alpha$ is the learning rate.
		\EndFor
	\end{algorithmic}
\end{algorithm}

\subsubsection{Adversarial Pixel Transformations}
\label{subsec: Adversarial Pixel Transformations}
For given input samples $\bm{x} \sim P_s$, we first find adversarial example, $\bm{x}' \;\; \text{subjected to:} \;\;  \Vert \bm{x} - \bm{x}' \Vert_{\infty} \leq \epsilon$, by maximizing the following objective ($\mathcal{L}_{max}$):
\begin{align}
\label{eq: attack_loss}
\underset{\bm{x}'}{\text{maximize}} \quad \mathcal{L}_{max} =  \|\mathcal{F} \left(\mathcal{T}_s(\bm{x}')\right)- \bm{x}\|^p,
\end{align}
where $\mathcal{T}_s$ represents the pixel transformation (\emph{e.g}\bmvaOneDot, rotation or jigsaw shuffle) that shifts the pixel positions. Therefore, our attack fools the model's ability to restore the transformed pixel space by maximizing the loss presented in Eq.~\ref{eq: attack_loss} and ultimately help to robustify self-supervised features. Our attack approach can also be extended to benefit from supervisory signals \emph{e.g}\bmvaOneDot, by fooling prototypes \cite{snell2017prototypical, li2020practical} as follows:
\begin{equation}
\label{eq: attack_loss_prototypical}
\underset{\bm{x}'}{\text{maximize}} \quad \sum^{C}_{c=1} \left(y_c \, \|\mathcal{F} \left(\mathcal{T}_s(\bm{x}')\right)- \bm{x}^{(c)}\|^p\right),
\end{equation}
where $C$ represents the number of categories, $y_c$ represents one-hot encoded labels, and $\bm{x}^{(c)}$ is the chosen prototype for a particular class. Our proposed attack objective in Eq.~\ref{eq: attack_loss_prototypical} helps to optimize for robust discriminative features with better adversarial transferability.

\subsubsection{Pixel Restoration}
\label{pixel_restoration}
For a given adversarial sample, $\bm{x}'$ created using Eqs. \ref{eq: attack_loss} or \ref{eq: attack_loss_prototypical},  we train the surrogate model, $\mathcal{F}$, by pixel restoration. Our loss function penalizes the model $\mathcal{F}$ by minimizing the reconstruction error between the original sample, $\bm{x}$, and the model's output for the transformed adversarial as well as the transformed original sample as follows:
\begin{equation}
\label{eq: out_loss}
    \mathcal{L}_{out} =\|\mathcal{F} \left(\mathcal{T}_s(\bm{x}')\right)-  \bm{x}\|^p + \|\mathcal{F} \left(\mathcal{T}_s(\bm{x})\right)-  \bm{x}\|^p.
\end{equation}

We further regulate the model's feature space during adversarial training by enforcing alignment between the original and adversarial feature distributions as follows:
\begin{equation}
\label{eq: feature_loss}
\mathcal{L}_{{feature}}=\|\mathcal{F}^{n} \left(\mathcal{T}_s(\bm{x}')\right)- \mathcal{F}^{n} \left(\mathcal{T}_s(\bm{x})\right)\|^p,
\end{equation}
where $\mathcal{F}^{n}$ represents the intermediate (encoder) layer output. Overall training objective is,
\begin{equation}
\label{eq: overall_loss}
    \mathcal{L}_{min} = \mathcal{L}_{out} + \lambda \mathcal{L}_{feature},
\end{equation}
where $\lambda$ is the balancing parameter.

\subsubsection{Behavior of Robust Loss Surfaces}
We visualize the loss landscapes of our trained autoencoders (Fig. \ref{Loss surfaces}). We use the filter normalization method proposed in \cite{li2018visualizing}, which shows the structure of the loss surface along with random directions near the optimal pretrained parameters. We observe that our approach leads to more flatter minima as compared to the baseline method \cite{li2020practical}. This has significant effect on finding generalizable adversarial example with better transferability (Sec.~\ref{subsec: Transferability from Few In-Domain Samples}).

\begin{figure}[t]
\begin{minipage}{0.69\linewidth}
    \centering

    \includegraphics[trim= 20mm 30mm 20mm 40mm, clip, height=2.9cm, width=0.49\textwidth]{images/Loss_Surface_Plots/Rotate.png}
    \includegraphics[trim= 20mm 30mm 20mm 40mm, clip,  width=0.49\textwidth]{images/Loss_Surface_Plots/Rotate_Ours.png}\\
    \vspace{-1em}
     \includegraphics[trim= 20mm 30mm 20mm 50mm, clip, width=0.49\textwidth]{images/Loss_Surface_Plots/Jigsaw.png}
    \includegraphics[trim= 20mm 30mm 20mm 50mm, clip, height=2.5cm, width=0.49\textwidth]{images/Loss_Surface_Plots/Jigsaw_Ours.png}
\end{minipage}
\hspace{0.01\linewidth}
\begin{minipage}{0.28\linewidth}
    \caption{\small Reconstruction loss landscape of autoencoders. The first column shows the loss landscapes of autoencoders trained using the baseline method on the task of rotation (\emph{top}) and jigsaw (\emph{bottom}). The second column shows the corresponding flatter loss surfaces obtained by training the autoencoders using our approach.}
    \label{Loss surfaces}
\end{minipage}
\end{figure}

 \section{Experimental Protocols}
 \label{sec: Experimental Protocols}

For a pixel range [0, 1], we create $l_\infty$ adversarial examples under perceptual budget of $\epsilon \le 0.1$ following \cite{li2020practical}. We show adversarial transferability of our approach against ImageNet trained models (Convolutional and ViTs), object detection and segmentation from in-domain and cross-domain (paintings, medical scans ) data samples. We used Adam \cite{kingma2014adam} optimizer with a learning rate of 0.001 for our proposed adversarial pixel restoration (Algo. \ref{algo: Adversarial Denoising and Restoration}) at $\epsilon \le \frac{2}{255}$, $\delta = \frac{2}{255}$, $\lambda=1$, $p=2$. We provide detailed ablative analysis on the effect of these hyper-parameters in Sec.~\ref{subsec: Ablative Analysis}.

\noindent\textbf{Surrogate Training  with Few Samples:} Similar to \cite{li2020practical}, we assume access to only few data samples (\emph{e.g}\bmvaOneDot, 20 samples to train a single autoencoder). We apply Eq. \ref{eq: attack_loss} as adversarial pixel transformation ($\mathcal{T}$) based on rotation or jigsaw in an unsupervised setting. We also incorporate our method in the supervised prototypical training of autoencoders mentioned in \cite{li2020practical}, were the reconstruction objective function used during the minimization and maximization step is formulated by Eq. \ref{eq: attack_loss_prototypical}. We compare the best prototypical setting reported in \cite{li2020practical}, comprising of multiple decoder networks. All models are trained for the same no. of iterations as \cite{li2020practical}. We provide pseudo code of Algo. \ref{algo: Adversarial Denoising and Restoration} in Appendix  \ref{appendix: pseudocode}.

\noindent\textbf{Surrogate Training with Large Dataset:}  There is abundance of unannotated data available via online sources. Therefore, we also scale our proposed self-supervised adversarial training to large-scale datasets. Specifically, we train a surrogate model (single autoencoder) on paintings (79k samples) \cite{paintings}, CoCo (40k samples) \cite{lin2014microsoft}, and Comics (50k samples) \cite{comics} to test unsupervised cross-domain adversarial transferability of our method.

\noindent\textbf{Target Models:}
Adversarial perturbations from our trained autoencoders (surrogate models) are transferred to classification models including convolutional: VGG-19 \cite{Simonyan2015}, Inception-v3 (Inc-V3) \cite{Szegedy2016}, ResNet-152 (Res152) \cite{He2016}, Dense161 \cite{Huang2017densely}, SeNet \cite{Hu2018}, Wide-ResNet-50 (WRN) \cite{Zagoruyko2016}, and MobileNet-V2 (MNet-V2) \cite{Sandler2018mobilenetv2}, and Vision Transformers: ViT-T and ViT-S \cite{dosovitskiy2020image}, DeiT-T and DeiT-S \cite{touvron2021training}. We also evaluated adversarial vulnerability of robust ResNet-50 models \cite{salman2020adversarially}. Further, we transfer attack to DETR \cite{carion2020end} and DINO \cite{caron2021emerging} to evaluate on object detection and video segmentation tasks.

\noindent\textbf{Evaluation Metrics:} We report drop in Top-1 (\%) accuracy, Mean Average Precision (mAP), and Jaccard Index for classification, object detection and segmentation, respectively.

\noindent\textbf{Datasets:} We evaluate classification models on 5k samples from ImageNet validation set in the same setting as \cite{li2020practical}. DETR and DINO are evaluated on CoCo (5k samples) and DAVIS (2k samples) validation sets respectively.

\noindent\textbf{Baseline Adversarial Attack:} For adversarial prototypical training, we use the same supervised attack objective as proposed by \cite{li2020practical} for direct comparison. Specifically, we use 200 iterations of I-FGSM \cite{kurakin2018adversarial} followed by 100 iterations of ILA \cite{huang2019enhancing}. The attack objective for the surrogate models trained in self-supervised manner is simply based on maximizing the reconstruction error as described in Eq. \ref{eq: attack_loss}.

\subsection{Results}

\subsubsection{Transferability from Few In-Domain Samples}
\label{subsec: Transferability from Few In-Domain Samples}
Our surrogate models trained on few data samples show significantly higher adversarial transferability as compared to the baseline \cite{li2020practical}. Attack generated on our self-supervised models (rotation, and jigsaw) performs even better than supervised (prototypical) models of \cite{li2020practical} (Tables \ref{tab:no-box table}, \ref{tab:no-box table_transformers}). Our approach further boosts the transferability rates when combined with supervised adversarial prototypical training. This compliments the benefits of our method in both supervised and self-supervised settings. We observe that vision transformers are more robust as compared to convolutional networks against such attacks \cite{naseer2021intriguing}, however, our approach provides non-trivial gains in fooling the Vision Transformers (Table \ref{tab:no-box table_transformers}). Similarly, adversarially robust models \cite{salman2020adversarially} are less vulnerable to such attacks (Fig. \ref{fig: transferability against robust models}).

\input{tables/inDomain_tables_classification}
\input{figures/inDomain_fig_classification_robust_models}
\input{tables/crossDomain_table_classification}
\input{tables/crossDomain_table_detection_segmentation}

\subsubsection{Transferability from Cross-Domain Samples}
Extra unsupervised data can boost the performance of adversarial training \cite{carmon2019unlabeled}. We extend our approach to large-scale, unlabelled datasets to observe its effect on adversarial transferability of our approach as explained below. All surrogate models are trained for 50 epochs for cross-task adversarial transferability experiments.

\noindent \textbf{Classification:} In this task, we train a single autoencoder and transfer its adversarial perturbations to ImageNet trained convolutional models as described in Sec.~\ref{tab: single_surrogate_classification_results}. We observe that our proposed adversarial training significantly improves upon the baseline \cite{li2020practical} (Table \ref{tab: single_surrogate_classification_results}).  We further note that the surrogate models trained on `paintings' dataset show higher adversarial transferability while `Rotation' as pixel transformation performs better. Further analysis on adversarial transferability of our attack is provided in Appendix \ref{appendix: results_eps_0.08}.

\noindent \textbf{Object Detection and Segmentation:} Our adversarial attack based on simple transformed reconstruction error (Eq.~\ref{eq: attack_loss}) compliments our proposed adversarial training and successfully fools DETR for object detection and DINO for video segmentation (Tables \ref{tab:DETR_results} \& \ref{tab:DINO_results}). This signifies that our attack can be launched in real-world setting without any knowledge about the deployed vision system.

\subsection{Ablative Analysis}
\label{subsec: Ablative Analysis}
We thoroughly analyze and develop better understanding about the behavior of our approach by studying the effect of its different components including, \textbf{a)} \emph{Effect of Perceptual Budget $\epsilon$ for our Single step Attack (Eq. \ref{eq: attack_loss})}, \textbf{b)} \emph{Effect of Iterative
Attack during Training}, \textbf{c)} \emph{Effect of Training Iterations and Data size}, and \textbf{d)} \emph{Contribution of Losses}. All ablative experiments are conducted in limited data setting (Sec.~\ref{subsec: Transferability from Few In-Domain Samples}).

\noindent \textbf{Effect of Perceptual Budget ($\epsilon$):} Our adversarial training (Algo. \ref{algo: Adversarial Denoising and Restoration}) is computationally efficient as it is based on a single attack (maximization) step. The effect of attack step size on model generalizability and hence its transferability is presented in Table \ref{tab: effect of epsilon}. We observe that adversarial perturbation computed on models trained with a smaller $\epsilon$ results in significant improvements in terms of attack transferability. This behaviour is in line with \cite{springer2021little} which shows that slightly robust classifier models generate highly transferable adversarial examples. This is further evident by the shift in attention caused by our method (Fig. \ref{fig:attn_vis}).

\noindent\textbf{Effect of Iterative Attack:} Increasing the attack iterations during our adversarial training reduces adversarial transferability, \emph{e.g}\bmvaOneDot, average Top-1 (\%) accuracy of convolutional models increases for attacks generated from autoencoders trained for 4 iteration (30.9\%) vs. single iteration (26.87\%) for rotation transformation. This is because attack becomes unsuccessful against highly robust autoencoders which leads to poor performance.

\noindent \textbf{Contribution of Losses:} We explore the effect of our proposed pixel and feature reconstruction losses (Eq. \ref{eq: overall_loss}) in Table \ref{tab: Loss_contrb}. We observe that feature reconstruction compliments the pixel reconstruction and leads to better surrogate model with more transferable adversarial space.

\noindent \textbf{Training iterations and Data Size:}  We explore the effect on transferability of surrogate models w.r.t \textbf{a)} training iterations, and \textbf{b)} the number of data samples in Fig. \ref{fig: ablation iterations and datasize}. Note that \cite{li2020practical} trains a single autoencoder on 20 samples and thus needs 250 autoencoder to attack 5k ImageNet validation samples. The number of autoencoders increases to 2.5k when only 2 samples are available for training of single model. In the same setting, the performance of our method improves with more iterations (Fig. \ref{fig: ablation iterations and datasize} \emph{left plot}) in contrast to \cite{li2020practical}. We report average Top-1 (\%) accuracy across all convolutional models (\emph{lower is better}). Similarly, as we increase the number of data samples during training and reduce the number of autoencoders, the performance of our approach significantly improves as compared to the baseline \cite{li2020practical}. This indicates that our adversarial objective successfully reduces overfitting while increases generalizability of crafted adversarial perturbations.

\input{tables/ablative_tables}

\begin{figure}[!t]
\small \centering
\begin{minipage}{0.68\textwidth}
    \begin{minipage}{.49\textwidth}
        \centering
        \includegraphics[width=\linewidth]{images/ablation/ab1.png}
    \end{minipage}
    \begin{minipage}{.49\textwidth}
        \centering
        \includegraphics[width=\linewidth]{images/ablation/ab2.png}
	\end{minipage}
\end{minipage}
\begin{minipage}{0.30\textwidth}
\vspace{-1em}
    \caption{\small Top-1 (\%) average accuracy against convolutional networks. The performance of our method improves with more training iterations and data size in contrast to the baseline \cite{li2020practical}.}
\label{fig: ablation iterations and datasize}
\end{minipage}
\vspace{-0.5em}
\end{figure}

\input{figures/attention_maps}

\section{Conclusion}
In this work, we show the benefits of adversarial training to learn transferable adversarial perturbations. Our approach trains an effective surrogate by learning to restore adversarial pixel transformations created via our proposed attack. Our adversarial training reduces overfitting during training and can exploit very few data samples to learn meaningful adversarial features while it can also scale to large unsupervised datasets. Our attack is task independent and allows cross-domain attacks (\emph{e.g}\bmvaOneDot, learning surrogate on comics and transferring its perturbations to models trained on natural images). Our results bring attention to the use of self-supervised adversarial training for transferable adversarial attacks.