\section{Introduction}
\label{sec:intro}
Multimodal learning has gained broad attention from computer vision community recently. For visual-lingual tasks, such as image captioning \cite{anderson2018vqamethod6, xu2015ic}, visual grounding \cite{yu2018vg}, and visual question answering (VQA) \cite{marino2019okvqa, vqa}, impressive progress has been made with powerful deep learning models. Compared to other multimodal tasks, VQA is a more challenging problem as it requires semantic understanding and detailed reasoning over the visual contents and textual elements. VQA for aerial images is a quite new task that has several potential real-world applications such as land use and disaster monitoring and environmental or urban planning \cite{rahnemoonfar2021floodnet}. However, the recent fine-tuning based methods \cite{lobry2020rsvqa} developed for natural scenes cannot handle the numerous difficulties for aerial images, including scale variation, object density, arbitrary orientation, and complex background. The numerous tiny objects in high-resolution aerial images require more specific descriptions in the questions to avoid ambiguity, and make the learning more difficult without any semantic supervision in VQA.

\begin{figure}[t!]
  \centering
  \begin{subfigure}[t]{0.45\linewidth}

  \includegraphics[width=1\linewidth]{image/example1.png}
  \subcaption*{{\textbf{$Q$:}} Are there more buildings than large vehicles in this image?\\ {\textbf{$Q$:}} How many small vehicles are there in this image?}
  \label{fig:example-1}
  \end{subfigure}
  \hfill
  \begin{subfigure}[t]{0.45\linewidth}

  \includegraphics[width=1\linewidth]{image/example2.png}
  \subcaption*{{\textbf{$Q$:}} What color is the second ship based on the left to right rule in this image?\\ {\textbf{$Q$:}} Where is the storage tank in this image?}
  \label{fig:example-2}
  \end{subfigure}
  \caption{Examples of image/question pairs from HRVQA. Ten types of questions are created, covering the inferences from common-seen task reasoning to specific attribute recognition.}
  \vspace{-5mm}
  \label{fig:examples}
\end{figure}

In addition to aforementioned difficulties, current research still lacks comprehensive well-annotated datasets, which are crucial to deep learning models.
Although there do exist some public datasets of remote sensing images for VQA \cite{lobry2020rsvqa,zheng2021mutual}, they are created with general questions and restricted in terms of spatial resolution, richness of concepts, complexity and diversity of questions, which make it hard to evaluate the real performance of these models. As it is so expensive to manually label the image/question pairs, human efforts need to be taken into account when creating a VQA dataset \cite{yi2019clevrer, zheng2021mutual}. Thus, a smart way of building a well-annotated dataset that contains not only general questions but specific details of the concepts is in high demand.

To advance the VQA research in computer vision and earth observation, we introduce a new dataset, HRVQA (High-Resolution aerial image Visual Question Answering). We collect 53512 aerial images and 1070240 QA pairs in total, which include 10 types of questions with respect to 27 category concepts. Given the quantity, diversity, and novelty of the proposed dataset, HRVQA shall be able to develop as a strong benchmark for aerial image VQA. As shown in Fig.~\ref{fig:examples}, these questions correspond to various scenes involving different concepts in aerial images.

Attention modules embedded in deep learning networks focus on the most relevant contents to improve performance for many computer vision tasks. They have been incorporated in VQA networks and also led to better performances for VQA \cite{dua2021vqamethod3, goyal2017method4, shih2016attentionmethod1}. Co-attention method \cite{kim2018banmethod7} parallelly derives the attended visual and textual features from individual regions and texts. Inspired by the transformers\cite{vaswani2017transformer}, deep modular co-attention networks \cite{yu2019mcan, aoa} address this problem by two units: self-attention (SA) and guided attention (GA). SA and GA extract intra-modal and cross-modal interactions with multi-head attentions, respectively. However, these attention units always calculate the weighted combination of features among the attended modalities, and ignore the positional information of the target concepts. This limitation leads worse results especially when asked a question about an isolated object with specific positional restrictions, which impairs the performance on aerial image VQA dealing with numerous tiny concepts.

In this paper, we propose GFTransformer, a modular gated co-attention network with a mutual fusion module. Inspired by ConVit \cite{d2021convit}, gated attention modules with positional embedding are employed to extend the previous modular co-attention network \cite{yu2019mcan}. Specifically, the positional embedding is first used to represent the spatial features when fed into a multi-head attention architecture. To make a balance between the content and positional term, a gated transformation is introduced on the combination of the attended features from both sides. By normalizing the gated results, we generate the focused features through multiple attention heads and remove the interference of the irrelevant attentions to achieve reliable reasoning.
Furthermore, a mutual fusion module is proposed to selectively extract the most useful information from the refined cross-modal features before predicting answers.
Our \textbf{main contributions} of this paper are summarized as follows:
\begin{itemize}
    \item To the best of our knowledge, HRVQA is the first high-resolution aerial image VQA dataset with diverse and concerning questions about not only overall information but specific attributes of various concepts. It can be used to evaluate the capabilities of VQA models on performing scene understanding and geo-spatial reasoning in aerial images.
    \item We propose an efficient labeling scheme for aerial image VQA dataset construction in a semi-automatic fashion, so that we can build a comprehensive well-annotated VQA dataset with a few manual efforts.
    \item In addition to the dataset, we also propose a novel model (GFTransformer) and benchmark several state-of-the-art VQA models, which could serve as the baseline for future research.
\end{itemize}

\section{Related Work}

\noindent
\textbf{Datasets for VQA.} Almost all the deep learning methods, especially strongly supervised algorithms, require a large amount of annotated data for training. Recently, there emerges a great interest in visual and textual context understanding tasks \cite{vqa,imagecaptioning,visualgrounding}, and many VQA datasets \cite{vg,yi2019clevrer,marino2019okvqa} have been introduced in computer vision. Among them, DAQUAR \cite{malinowski2014daquar} is the first VQA dataset designed as a benchmark, which is built with the images of indoor scenes and questions about the basic information of various furniture. VQA 2.0 \cite{goyal2017method4}, one of the most widely used VQA datasets, focuses on both wild-world and indoor scenes for various questions with the images from COCO \cite{lin2014coco}. GQA \cite{hudson2019gqa} presents natural images with scene graphs and questions requiring diverse reasoning skills and sequential steps to generate answers. With respect to the field of earth observation, there have also been some pioneers delving into VQA \cite{rahnemoonfar2021floodnet,yuan2022easy}. RSVQA \cite{lobry2020rsvqa} consists of one low-resolution subset and one high-resolution subset with simple QA pairs. RSIVQA \cite{zheng2021mutual} is composed of available image classification and object detection datasets with restricted QA pairs generated based on the annotations. Differently, our HRVQA focuses on the intelligent perception and reasoning in high-resolution aerial images with more diverse questions.

\vspace{2mm}
\noindent
\textbf{QA Pair Generation.} Question Generation (QG) is a quite active research field in natural language processing \cite{du2017learningqg1,lopez2020transformerqg2}, which could be used as a preparation stage for other textual tasks, including question answering \cite{duan2017qgqa1} or information retrieval \cite{mass2020unsuperviseqgir}. In computer vision, it aims at generating QA pairs for a given image (or video) as training data or means of data augmentation for VQA \cite{li2018vg4qaonly333}. In this paper, our QA pair generation is based on a template strategy that requires the corresponding collection of visual information to answer the generated questions.

\vspace{2mm}
\noindent
\textbf{VQA Methods.}
In the last decade, we have witnessed the rising and fast pace developments of deep learning models for VQA \cite{goyal2017method4,wang2015vqamethod1,kim2018banmethod7,zhou2020method13,yu2019mcan}. In the general framework for VQA, a model needs to process the input image and question as global features separately and then fuse them together with a multimodal fusion module to predict the answers. RSVQA \cite{lobry2020rsvqa} included a feature extraction part based on ResNet-152 \cite{he2016deepresnet} and skip-thought \cite{kiros2015skip}, a point-wise multiplication layer for fusion and a classifier for prediction. However, only global features may mislead the learning process for some questions requiring local information. Attention techniques can handle the shortcoming by focusing on the most relevant visual regions or textual phases. SAN \cite{yang2016stackedsan} proposed a stacked attention network to learn the relationships among targets, which helps to discard the irrelevant objects in the image. MCAN \cite{yu2019mcan} adopted a dense co-attention model to learn the textual attention for questions and the visual attention for images simultaneously. Martins et al. \cite{martins2020sparse} and Zhou et al. \cite{zhou2021trar} followed \cite{yu2019mcan} and exploited different compact region selection methods to extract valid features. Whitehead et al. \cite{whitehead2022reliableeccv2022} utilized a multimodal selection function to estimate the correctness of the answers predicted by a co-attention model. One limitation of the aforementioned co-attention models is that the equally treated feature representations extracted from self-attention modules are directly projected into the answer space without further differentiation. Differently, our GFTransformer investigates the weighted information for each query in both modalities to improve the co-attention architecture, and selects the attended features for predictions.
\section{Construction of HRVQA Dataset}
In general, HRVQA dataset contains 1070240 QA pairs and 53512 images from high-resolution aerial scenes. In this section, we first introduce the resources of images and then explain the process of question and answer generation, respectively. Finally, we analyze the visual and lingual properties of the proposed dataset.

\subsection{Image Collection}
To build a VQA dataset with aerial images, there are several unique visual features to be taken into account. For aerial images, the (spatial) resolution is an extremely important property for many tasks, which indicates how detailed the ground contents can be observed. Besides, the images in various scenes need to contain diverse concepts. The aerial images of nadir view need to show the ground targets clearly.

Given that the current public aerial image VQA datasets \cite{lobry2020rsvqa,rahnemoonfar2021floodnet,zheng2021mutual} cannot meet the aforementioned requirements, we collect high-resolution aerial images from Dutch OpenData\cite{quickortho} for our VQA visual part. It is a collaboration of Dutch governmental organizations with the aim of providing nationwide aerial images in a ground pixel resolution of 8 cm. After selecting 4 typical cities in Netherlands, \textit{Utrecht, Rotterdam, Enschede and Amsterdam}, we obtain 53512 nadir-viewed aerial images of $1024 \times 1024$ pixels that contain rich and different concepts in various scenes (e.g., industry, harbor, station, sports)
\subsection{Question Generation}
\label{qg}
As it is tedious to completely include all the questions that one could ask for a given aerial image, our question set construction starts with the most related and common ones. To generate richer and more diverse questions, we consider the construction from three aspects: category concepts, question types and templates. We randomly generate 200 questions for each image to avoid missing of the questions about relatively rare concepts (e.g., plane).

\noindent
\textbf{Category Concepts.} We choose 27 category concepts~\footnote{The examples for these concepts are provided in the supplementary.} (including isolated objects and region stuff), \textit{small-vehicle, large-vehicle, ship, plane, storage tank, baseball diamond, tennis court, basketball court, bridge, helipad, helicopter, soccer ball field, swimming pool, roundabout, ground track field, harbor, airport, track, outdoor fitness field, high-density urban area, low-density urban area, rural area, industrial area, station, sports area, water section, building}. These concepts are selected according to the frequency of occurrence and the usage in our daily activities, which support VQA system to serve for more potential real-world applications. We further introduce two common attributes, \textit{size} and \textit{location}, for every concept, and two special attributes, \textit{shape} and \textit{color}, only for isolated objects. Each attribute is an optional restriction on the concepts to make the reasoning more detailed.

\noindent
\textbf{Question Types.} The aim of a question lies on the information that a questioner wants to know from the image. Different from the simple classification of question types for natural images, \textit{Yes/No, Number} and \textit{Others}, we further classify the questions to be generated into 10 types according to their functions, \textit{Number, Yes/No, Areas, Location, Color, Shape, Size, Scene, Transportation} and \textit{Sports}. They are closely related to the selected concepts and able to describe various aspects of them (from general to specific information). Note that it is extremely vital to avoid ambiguity for some question types in VQA due to the fact that there is always more than one object with the same category in aerial images. To improve the clarity of the questions, we add a set of more specific descriptions on the related concepts according to the relevant spatial location. For instance, we ask \textit{What color is the second ship in the image based on the left to right rule?} rather than \textit{What color is the ship in the image?} This solves the problem that the questions about isolated objects cannot be asked in the current remote sensing image VQA datasets \cite{lobry2020rsvqa,zheng2021mutual}.

\noindent
\textbf{Question Templates.} Given the randomly selected concept and question type from the corresponding candidate pools, respectively, the main question structure starting with \textit{How, What, Where, Is, Are} and \textit{Can} is defined. Then a relevant template for each question type is randomly selected from a pre-defined template set. For instance, we could ask \textit{How many ships are there in this image?} or \textit{What is the number of ships in this image?} for a \textit{Number} question.
\subsection{Answer Generation}
To obtain the answers to the generated questions, we collect semantics of the involved concepts from images based on an object detection model \cite{yolov5} and the available annotations from Dutch open resource PDOK \cite{pdok}, and then match them with the functions of the proposed questions.

Specifically, DOTA \cite{dota, dota-v2} contains a small set of aerial images as well as bounding box annotations for 18 category objects, which cover the majority of the concepts in our datasets. Given the advances of object detection models pre-trained on large-scale datasets, we utilize a detection model YOLO v-5 \cite{yolov5} and finetune it on the subset to predict the objects in our aerial images. Achieved 78.7\% mAP detection results could hugely alleviate the burdens of manual editing (e.g., tiny and frequent objects like \textit{vehicles, ships}). PDOK \cite{pdok} provides open datasets as well as recent metadata in the form of point lists for each ground target of several categories. Then we extract the spatial information about \textit{tracks, buildings} and \textit{water sections}. Upon the collection of all the semantics is completed, we generate the answers to the questions based on the spatial and category information. Note that we set a flag $\in \{0, 1\}$ for every triplet to indicate its relevance. There are two steps for calculating the relevance: first checking if the image contains the target concept involved in the question, then judging if the attribute of the concept in the image is consistent with the contents in the question.

\begin{figure*}[ht]
  \centering
  \begin{subfigure}[t]{0.35\linewidth}

    \includegraphics[width=1\linewidth]{image/question_answer_distribution3.png}
    \caption{Distribution of Answers Per Question Type.}
    \label{fig:short-a}
  \end{subfigure}
  \hfill
  \begin{subfigure}[t]{0.3\linewidth}

    \includegraphics[width=1\linewidth]{image/question_length.png}
    \caption{Distribution of Question Lengths.}
    \label{fig:short-b}
  \end{subfigure}
  \hfill
  \begin{subfigure}[t]{0.3\linewidth}

    \includegraphics[width=1\linewidth]{image/question_type1.png}
    \caption{Distribution of Question Types.}
    \label{fig:short-c}
  \end{subfigure}

  \caption{Statistics of question/answer pairs in HRVQA. \textbf{Left} represents distribution of answers per question type in the following order: \textit{Number, Yes/No, Areas, Size, Location, Color, Shape, Sports, Transportation and Scene}. The short names for the answers are defined as: \textit{A0-$0m^2$, A2-$between\ 10\ and\ 100m^2$, A3-$between\ 100\ and\ 1000m^2$, DNE-does not exist, TL-top left, BR-bottom left, OE-other exercise, OF-outdoor fitness, SW-swimming, HDU-high-density urban area, RA-rural area, HA-harbor, LDU-low-density urban area, and SA-sports area}.}
  \label{fig:short}
\end{figure*}

We also apply some transformations on the numerical answers inspired by \cite{lobry2020rsvqa}. Differently, only answers to \textit{Areas} question are quantized into the following categories: a) $0m^2$; b) $between\ 0\ and\ 10m^2$; c) $between\ 10\ and\ 100m^2$; d) $between\ 100\ and\ 1000m^2$; e) $more\ than\ 1000m^2$. We insist to predict exact values for the \textit{Number} questions because they are more meaningful and challenging for VQA.
\subsection{Triplet Filter}
\label{triplet filter}
To maintain the balance and diversity of the proposed dataset, we further adjust the distribution of different types of questions and remove the majority of the triplets with irrelevant answers, which mean the meaningless short-cuts whose flags equal 0 in the previous settings. The first step is to reduce the proportion of meaningless triplets. Specifically, among the randomly generated 200 triplets, we reassign 20 QA pairs with 70\% active flags for each image (if not satisfied, repeat the generation scheme until 70\%). Then we replace the half of the questions from transportation and scene types (easy to predict) with new randomly generated questions from other types (hard to predict).

Till this end, a balanced and diverse aerial image VQA dataset has been created. The proposed method for QA pair generation enables the semi-automatic and scalable VQA triplet construction with only a few manual efforts.

\subsection{Properties of HRVQA Dataset}
The images in HRVQA are all from the same airborne source provided by \cite{quickortho}. The collection of the images took place in the leafless season (from mid-February to mid-April), so that as much as possible information can be captured under the trees and vegetation from the images. We collect the RGB-channel aerial images in 8 cm resolution from 4 typical Dutch cities. Table \ref{tab:statistics of images} shows the statistics of images and concepts in terms of the number and covering area. Note that we do not present the covering area for the chosen concepts due to the uncertainty of region stuff’s boundary.

\begin{table}
  \caption{Statistics for images and concepts in HRVQA.}
  \centering
  \setlength\tabcolsep{2.5pt}
  \begin{tabular}{p{1.85cm}cccc}
    \toprule
    Number & Utrecht & Rotterdam & Enschede & Amsterdam\\
    \midrule
    \# of images & 12320 & 14716 & 6762 & 19714\\
    \# of concepts & 141033 & 206306 & 71705 & 236152\\
    Area ($km^2$) & 100 & 110 & 72 & 156\\
    \bottomrule
  \end{tabular}
  \label{tab:statistics of images}
  \vspace{-5mm}
\end{table}
Moreover, we show the statistics of the language distribution in the proposed dataset. In general, HRVQA covers 10 interesting and meaningful types of questions requiring scene understanding and reasoning inference. Fig. \ref{fig:short-a} presents the distribution of answers per question type. We can see that some question types, such as \textit{Size, Shape}, and \textit{Transportation} are dominated by one or two answers due to the real circumstance under the bird's view (e.g., roundabouts are always round; car is the most common transportation in the cities). Other question types are more even in answers. We further show the distribution of question lengths counted by every single word in Fig. \ref{fig:short-b}, and $9.7$ is the average value for our dataset. Fig. \ref{fig:short-c} indicates the distribution of question types, which indirectly implies the distribution of different questions about general and detailed information.
\section{Method}
\label{method}
To deal with high-resolution aerial image VQA, we propose GFTransformer (Gated attention and mutual Fusion Transformer), as illustrated in Fig.~\ref{fig:pipeline}. We first introduce the input feature representations from images and questions, and explain self-attention and guided-attention units adopted in our baseline model \cite{yu2019mcan}. Then we propose a gated attention module to extract the attended features with positional information. By cascading a number of aforementioned layers in depth, an encoder-decoder model refines the attended intra-modal and cross-modal features. Finally, a mutual fusion module is proposed to selectively combine the attended visual and lingual features together before projected into answer space.

\begin{figure*}[ht]
  \centering
  \includegraphics[width=0.85\linewidth]{image/pipeline.png}
  \caption{Pipeline of the proposed GFTransformer. The different units in the Encoder-Decoder layers represent self multi-head attentions (S) and guided multi-head attentions (G) with gated positional embeddings. With the Mutual Fusion module, the attended features are selectively combined to predict a desired answer.}
  \label{fig:pipeline}
  \vspace{-5mm}
\end{figure*}

\vspace{2mm}
\noindent
\textbf{Input Feature Representations.} Given that VQA models develop cross-modal mathematical operations on multi-source data, the images and questions need to be represented as vectors by using different feature extraction methods, respectively. \cite{anderson2018vqamethod6} first represents the images as a set of regional features in a bottom-up and top-down mode, which is powerful for the natural scenes and thus adopted in many VQA models \cite{anderson2018vqamethod6,zhou2020method13,zhou2021trar}. In contrast to natural images, aerial images cover numerous tiny objects in some cases (e.g., vehicles in a parking lot), which impair the capability of regional representations. To overcome this limitation, we replace the regional visual features in our baseline model \cite{yu2019mcan} with grid features $X$ generated by ResNet-152 \cite{he2016deepresnet}. Following \cite{yu2019mcan}, questions are represented into 512-D vectors by the GloVe word embeddings \cite{pennington2014glove} followed a LSTM \cite{hochreiter1997lstm} layer, to generate the question features $Y$.

\vspace{2mm}
\noindent
\textbf{Transformer Units.} Multi-head attention layers parallelly develop $N$ heads to capture different focuses on the features, where each head could be conducted by a scaled dot product attention function. The standard transformer attention function is shown as follows:
\begin{equation}
    f_{trans} = A(Q,K,V) = {\rm Softmax}(\frac{QK^T}{\sqrt{d}})V,
\end{equation}
where $Q,K,V$ indicate the queries, keys, and values, respectively, and $d$ represents the dimension of keys and values (normally set the same number for both). The function outputs the weighted summation over the values with attended information obtained from keys and queries. Specifically, the results of dot products between queries and keys are divided by $\sqrt{d}$ as attentions, and then the attentions are normalized to generate new weighted values.

Self-attention unit (SA) and guided-attention unit (GA) are built on the top of the multi-head attentions followed a feed-forward network (FFN) with ReLU activation and dropout. The residual connection \cite{he2016deepresnet} is adopted to maintain the reliable information and address gradient issues. The main difference between these two units is that SA only takes the input features from one modality for the intra-modal learning (word-to-word and region-to-region) while GA learns to capture the cross-modal relationships (word-to-region, region-to-word). Specifically, GA takes the queries from one modality to guide the attention learning based on the keys and values from the other one while SA only handles the individual elements.

\vspace{2mm}
\noindent
\textbf{Gated Attention Module.} The aforementioned attention units capture the relations between $Q$ and $K,V$, and output the weighted features without discriminate selection, which may mislead the answering process. To overcome the limitation, we add a gated attention module based on a positional guidance. In detail, the positional encoding $r_{xy} $ is used to represent the relative position (distance representations) of patches $x$ and $y$ among grid regions, and the corresponding embedding $P$ for this relation could be further trained with attention heads. We fix the relative positional encodings and make the network only learn the embedding to discover the adaptive attention span \cite{sukhbaatar2019adaptivespan}. The learned attention is shown as follows:
\begin{equation}
    f_{atte} = {\rm Softmax}(\frac{QK^T}{\sqrt{d}}+P^Tr_{xy}),
\end{equation}
As the magnitudes of both terms always face the issue of mismatching, ${\rm Softmax}$ cannot take the content and positional information into account perfectly at the same time. To alleviate the instability of the learning process, we adopt a learnable parameter $\theta$ in each attention head as the factor to balance the combination of both sides, which is a useful strategy to measure the importance of different terms \cite{d2021convit}. With normalization on the summation of both terms, the gated positional attention is completed as follows:
\begin{equation}
    f_{atte} = {\rm Normalize}((1-\sigma(\theta))f_c+\sigma(\theta)f_p),
\end{equation}
\begin{equation}
    f_c = {\rm Softmax}(\frac{QK^T}{\sqrt{d}}),   f_p = {\rm Softmax}(P^Tr_{xy}),
\end{equation}
where $\sigma$ is a sigmoid function. We initialize $\theta=1$ for all the heads and layers in the transformer architecture. Fig. \ref{fig:gated attention} shows the architecture of the gated attention unit.

\begin{figure}[t]
  \centering
  \includegraphics[width=0.57\linewidth]{image/gated_units1.png}
  \caption{Gated attention unit with multi-head attention (MHA). \textit{X} and \textit{Y} denote the visual and lingual features, respectively.}
  \label{fig:gated attention}
  \vspace{-5mm}
\end{figure}

Till this end, we feed the visual features $X$ and question features $Y$ into an encoder-decoder transformer with cascaded gated attention units. Simply, the output of the last attention layer is used as the input for the next one. We refine the features by stacking 6 gated attention units for intra-modal and cross-modal leaning in our encoder-decoder transformer~\footnote{More details about the architecture can be found in the supplementary.}.

\vspace{2mm}
\noindent
\textbf{Mutual Fusion Module.}
Yu et al. \cite{yu2019mcan} uses a simple multimodal fusion model based on a two-layer MLP (fully connected layer, ReLU, and dropout) to fuse the individual attended features, which is modal-agnostic. In this paper, we propose a mutual fusion module to project the features from both modalities into a better output space. First, the same MLP is adopted to generated attended features $X'$ and $Y'$, respectively. Different from \cite{yu2019mcan}, we use the features in one modality to guide the refining of the features from the other. Specifically, we summarize $X'$ and $Y'$ followed by a three-layer MLP (fully connected layer and dropout). To determine the impact of information from each modality, we add a ${\rm Softmax}$ on the summation and generate new mutual weighted features $X''$ and $Y''$ with the similar operations for $X'$ and $Y'$, respectively. With two linear projection matrices on $X''$ and $Y''$ followed a ${\rm LayerNorm}$ layer for stable training, we obtain the fusion results for the final predictions, which is realized by a sigmoid activation. Fig. \ref{fig:mutual fusion} illustrates the architecture of the proposed mutual fusion module.

\begin{figure}[t]
  \centering
  \includegraphics[width=0.8\linewidth]{image/fusion.png}
  \caption{Mutual fusion module after the Encoder-Decoder. Summation operation followed a \textit{MLP} and a ${\rm Softmax}$ helps to select the attended features from both modalities, to generate fusion results \textit{F}.}
  \label{fig:mutual fusion}
  \vspace{-5mm}
\end{figure}

\section{Experiments}
In this section, we conduct experiments to verify the effectiveness of our GFTransformer on the proposed HRVQA dataset. We first present the setup for the experiments, and then perform a benchmark based on several state-of-the-art VQA models. To further analyze the effectiveness of core model designs, we conduct extended ablation studies. Finally, comparative experiments on RSVQA-HR \cite{lobry2020rsvqa} are completed to validate the robustness of our model.

\begin{table*}
  \caption{Benchmark results of type-wise accuracy on HRVQA. The short names for the question types are defined as: $Y/N-Yes/No, Locat.-Location, Trans.-Transportation$. Note that $\dagger$, $\ddagger$, and $\ast$ represent basic CNN models, attention based CNN models, and transformers, respectively.}
  \centering
  \setlength\tabcolsep{5.5pt}
  \begin{tabular}{p{2cm}cccccccccccc}
    \toprule
    Method & Number & Y/N & Areas & Size & Locat. & Color & Shape & Sports & Trans. & Scene & OA & AA \\
    \midrule
    SAN\cite{yang2016stackedsan}$\ddagger$   & 62.89 & 82.15 & 86.30 & 93.48 & 60.19 & 61.97 & 92.13 & 72.86 & 87.72 & 74.84 & 79.01 & 77.45 \\
    MUTAN\cite{ben2017mutanmethod10}$\ddagger$  & 62.05 & 87.44 & 83.43 & 93.41 & 59.75 & \textbf{70.66} & 87.58 & 61.59 & 82.51 & 73.96 & 77.97 & 76.24 \\
    MCAN\cite{yu2019mcan}$\ast$   & 66.53 & 93.17 & 97.88 & 93.27 & 60.52 & 44.04 & 96.44 & \textbf{78.31} & 88.71 & 77.68 & 80.69 & 79.66 \\
    RSVQA\cite{lobry2020rsvqa}$\dagger$ & 61.97 & 84.81 & 75.50 & 93.41 & 63.96 & 67.85 & 87.05 & 62.34 & 81.82 & 67.98 & 76.19 & 74.67 \\
    M-CA\cite{martins2020sparse}$\ast$   & 66.25 & 93.22 & 97.93 & 93.30 & 61.23 & 44.48 & 96.46 & 77.28 & 88.71 & 77.92 & 80.75 & 79.68 \\
    TRAR\cite{zhou2021trar}$\ast$   & 66.86 & 93.31 & 97.26 & 93.47 & 63.20 & 44.50 & 96.51 & 76.76 & 88.84 & \textbf{78.69} & 81.02 & 79.94\\
    AOA\cite{aoa}$\ast$ & \textbf{66.97} & \textbf{93.41} & \textbf{97.98} & 93.55 & 62.20 & 44.83 & 96.37 & 77.70 & 88.68 & 78.12 & 80.98 & 79.98 \\
    FETH\cite{yuan2022easy}$\dagger$ & 62.15 & 87.86 & 82.37 & 93.44 & 62.88 & 68.13 & 90.59 & 72.93 & 84.23 & 77.07 & 79.49 & 78.17 \\
    Swap-m\cite{gupta2022swapmix}$\ast$ & 63.98 & 89.60 & 94.56 & 90.05 & 59.07 & 43.42 & 92.81 & 76.34 & 85.28 & 77.61 & 78.26 & 77.27 \\
    GFT (ours)$\ast$   & 66.50 & 93.32 & 97.11 & \textbf{93.72} & \textbf{74.82} & 45.36 & \textbf{96.67} & 77.03 & \textbf{88.87} & 77.30 & \textbf{81.71} & \textbf{81.07}\\
    \bottomrule
  \end{tabular}
  \label{tab:results in vhrqa}
  \vspace{-5mm}
\end{table*}
\subsection{Experiment Setup}
\label{setup}
\noindent
\textbf{Datasets.} To evaluate the proposed method, the main experiments are conducted on the proposed high-resolution aerial image dataset, HRVQA. To fairly compare the performances of different models designed for natural scenes with ours, we re-trained these models on HRVQA. We split the dataset into a training set (\textit{Utrecht} and \textit{Rotterdam}), a validation set (\textit{Enschede}), and a test set (\textit{Amsterdam}), and their proportions are 50.5\%, 12.6\%, and 36.9\%, respectively. To further verify the generalization and robustness of the proposed method, we conducted extensive experiments on another aerial image dataset RSVQA-HR\cite{lobry2020rsvqa} that contains a considerable number of image/question/answer triplets.

\vspace{2mm}
\noindent
\textbf{Implementation Details.} To make a fair comparison, we conducted all the experiments on the same \textit{PyTorch} environment with 1 NVIDIA RTX 2080 Ti GPU in 11GB memory, and trained the models by using the \textit{Adam} optimizer with $\beta_1=0.9$ and $\beta_2=0.98$ for 13 epochs. We set the base learning rate to the minimum of 0.0000025 and 0.0001 and decay it by 0.2 at epoch 10 and 12. The batch size is set to 32, and the latent dimension in the multi-head units, the number of heads, and the latent dimension for each head are set to 512, 8, 64, respectively. More details can be found in the supplementary material. We used the mainstream overall accuracy (OA) and average accuracy (AA) for the evaluation statistics.

\subsection{Benchmark Results}
To promote the development of VQA in earth observation, we built a benchmark by conducting experiments on the proposed HRVQA over different methods, including basic CNN models, attention based CNN models, and transformer models. Besides, we provided a new baseline model for aerial image VQA, and compared the results with others. The comparative results on the test set are listed in Table \ref{tab:results in vhrqa}. We observed the following benchmark results: (1) Our proposed method achieves the best overall accuracy (81.71) and average accuracy (81.07) on HRVQA. (2) Different methods are proposed for various directions in VQA, based on their specially designed modules, training schemes and so on. This leads their performances differ in individual question types. For instance, AOA \cite{aoa} handles the common-seen question better than the others (e.g., \textit{Number} and \textit{Yes/No} are classical questions for natural images), while our method performs better in the positioning related questions (e.g., \textit{Size, Location, Shape}). (3) CNN models \cite{lobry2020rsvqa, yuan2022easy, yang2016stackedsan, ben2017mutanmethod10} incline to perform better than transformers \cite{aoa,yu2019mcan, zhou2021trar} in the term of \textit{Color} questions, which is nearly 25\% gap. On the other hand, these questions are always about the information of isolated objects (e.g., \textit{small vehicle, large vehicle, ship}), which are significantly smaller than the others. We conjecture that convolution and pooling operations are more effective in capturing the color attribute of tiny objects from aerial images when the supervision of corresponding semantics is absent in VQA.

\begin{figure*}[ht]
  \centering
  \begin{subfigure}[t]{0.45\linewidth}

    \includegraphics[width=1\linewidth]{image/vis-m1.png}
    \subcaption*{\textbf{$Q$:} What sport can people do in this image? \textbf{$A$:} playing soccer}
  \end{subfigure}
  \hspace{8mm}
  \begin{subfigure}[t]{0.45\linewidth}

    \includegraphics[width=1\linewidth]{image/vis-m2.png}
    \subcaption*{\textbf{$Q$:} What is the color of the third large vehicle based on the left to right rule in this image? \textbf{$A$:} grey}
  \end{subfigure}
  \vspace{-2mm}
  \caption{Visualizations of results from GFTransformer on HRVQA. \textbf{Right} side of each pair shows the attentions across bounding boxes, and the region in a green box means the highest weight included.}
  \label{fig:visualization}
  \vspace{-2mm}
\end{figure*}

\begin{table*}
  \caption{Ablation studies for individual modules. G-MHA and MF represent gated attention unit and mutual fusion module, respectively.}
  \centering
  \setlength\tabcolsep{5.5pt}
  \begin{tabular}{p{2.2cm}cccccccccccc}
    \toprule
    Method & Number & Y/N & Areas & Size & Locat. & Color & Shape & Sport & Trans. & Scene & OA & AA \\
    \midrule
    Baseline\cite{yu2019mcan}   & 66.53 & 93.17 & 97.88 & 93.27 & 60.52 & 44.04 & 96.44 & 78.31 & 88.71 & 77.68 & 80.69 & 79.66\\
    +\textit{only} G-MHA & 66.57 & 93.24 & 97.55 & 92.89 & 74.15 & 44.72 & 96.63 & 76.59 & 88.15 & 77.66 & 81.37 & 80.81 \\
    +\textit{only} MF & \textbf{66.84} & \textbf{94.34} & \textbf{98.22} & 93.40 & 61.12 & \textbf{45.76} & 96.39 & \textbf{78.44} & 88.87 & \textbf{78.03} & 81.07 & 80.04\\
    GFT (ours)   & 66.50 & 93.32 & 97.11 & \textbf{93.72} & \textbf{74.82} & 45.36 & \textbf{96.67} & 77.03 & \textbf{88.87} & 77.30 & \textbf{81.71} & \textbf{81.07}\\
    \bottomrule
  \end{tabular}
  \label{tab:ablation1 in vhrqa}
  \vspace{-2mm}
\end{table*}

In summary, the proposed benchmark comprehensively introduces a challenging problem about VQA for aerial images. It contains not only the common questions but also the new designed questions for aerial images. Moreover, the proposed method outperforms the state-of-the-art models in overall accuracy and average accuracy, respectively. Fig. \ref{fig:visualization} shows some examples of our predictions and attention across bounding boxes.

\subsection{Ablation Studies}
\label{ablation}
To verify the effectiveness of the proposed individual modules and evaluate different input representations, we design a number of ablation studies. More information can be found in the supplemental material.

\vspace{2mm}
\noindent
\textbf{Effectiveness of Proposed Modules.} In our proposed GFTransformer, there are two core improvements over the baseline model: gated attention unit (G-MHA) and mutual fusion module (MF). G-MHA is present to add positional attentions on the features while MF tries to combine the attended features from both modalities together. Table \ref{tab:ablation1 in vhrqa} shows the results of individual module added on the baseline model \cite{yu2019mcan}. When only developing MF, the overall accuracy is improved by 0.38\%. And 0.68\% improvement is achieved with only G-MHA. Moreover, when combining the two components together, we get the best overall accuracy 81.71\% and average accuracy 81.07\%, which indicates that they could complement each other.

\vspace{2mm}
\noindent
\textbf{Different Input Feature Representations.} To figure out the performances of different feature representations, we further ran four groups of combinations: region features + GloVe features \cite{pennington2014glove}, region features + Bert (a pre-trained model in machine translation) features \cite{devlin2018bert}, grid features + GloVe features \cite{pennington2014glove}, and grid features + Bert features \cite{devlin2018bert}. The results are listed in Table \ref{tab:ablation3}. Grid features are more suitable for aerial image VQA due to the resolution and the size of concepts. Pre-trained Bert \cite{devlin2018bert} does not improve the results when dealing with the questions from HRVQA.

\begin{table}
  \caption{Ablation studies for different input feature representations on HRVQA.}
  \centering
  \setlength\tabcolsep{12pt}
  \begin{tabular}{p{3cm}cc}
    \toprule
    Method & OA & AA \\
    \midrule
    Region + GloVe\cite{pennington2014glove}  & 80.78 & 79.25 \\
    Region + Bert\cite{devlin2018bert}  & 80.73 & 79.43 \\
    Grid + Bert\cite{devlin2018bert}  & 81.25 & 80.63 \\

    Grid + GloVe\cite{pennington2014glove}    & \textbf{81.71} & \textbf{81.07} \\
    \bottomrule
  \end{tabular}
  \label{tab:ablation3}
  \vspace{-5mm}
\end{table}
\subsection{Results on RSVQA}
To further evaluate the robustness of the proposed method on various datasets, we trained and tested different VQA models on the high-resolution subset (RSVQA-HR \cite{lobry2020rsvqa}) of the available aerial image dataset RSVQA. RSVQA-HR has two test sets called Test-1 and Test-Phila. Test-1 contains the similar regions as the training samples while Test-Phila covers the regions from another city called Philadephia, which are not seen during the training process. Table \ref{tab:results in rsvqa test phili} shows the comparative experimental results on RSVQA-HR Test-Phila.
We could see the significant improvements with our proposed GFTransformer. Specifically, compared to the state-of-the-art VQA model designed for aerial images \cite{yuan2022easy}, 2.45\% overall accuracy and 2.70\% average accuracy boosts are obtained on Test-Phila. Besides, we perform better than the models designed for natural scenes \cite{aoa,ben2017mutanmethod10,yang2016stackedsan,zhou2021trar,martins2020sparse} in general with 1.53\% overall accuracy and 1.74\% average accuracy improvements.

\begin{table}
    \caption{Results of type-wise accuracy on RSVQA~\cite{lobry2020rsvqa} Test-Phila.}

  \centering
  \setlength\tabcolsep{2.75pt}
  \begin{tabular}{p{2cm}cccccc}
    \toprule
    Method & Pres & Count & Comp & Area & OA & AA \\
      & -ence &  & -arison &  &  &  \\
    \midrule
    SAN\cite{yang2016stackedsan}$\ddagger$  & 87.95 & 61.24 & 87.83 & 77.98 & 79.48 & 78.75 \\
    MUTAN\cite{ben2017mutanmethod10}$\ddagger$ & 87.50 & \textbf{62.47} & 88.39 & 79.75 & 80.13 & 79.53 \\
    MCAN\cite{yu2019mcan}$\ast$   & 87.73 & 62.28 & 86.41 & 83.12 & 79.97 & 79.88 \\
    RSVQA\cite{lobry2020rsvqa}$\dagger$ & 86.26 & 61.47 & 85.94 & 76.33 & 78.23 & 77.50 \\
    M-CA\cite{martins2020sparse}$\ast$   & 87.54 & 61.63 & 86.94 & 84.64 & 80.15 & 80.19 \\
    TRAR\cite{zhou2021trar}$\ast$   & 87.11 & 61.92 & 87.27 & 86.28 & 80.47 & 80.65 \\
    AOA\cite{aoa}$\ast$   & \textbf{89.08} & 61.26 & 88.27 & 86.04 & 81.11 & 81.17 \\
    FETH\cite{yuan2022easy}$\dagger$ & 87.97 & 61.95 & 87.68 & 78.62 & 79.29 & 79.06 \\
    GFT (ours)$\ast$   & 89.00 & 61.71 & \textbf{89.56} & \textbf{86.77} & \textbf{81.74} & \textbf{81.76} \\

    \bottomrule
  \end{tabular}
  \label{tab:results in rsvqa test phili}
  \vspace{-5mm}
\end{table}

\section{Conclusion}
In this paper, we propose a new VQA dataset for high-resolution aerial images, HRVQA, to build a comprehensive benchmark for computer vision and earth observation research. We introduce a semi-automatic construction scheme for labeling the image/question pairs with 10 concerning types of questions. Moreover, we provide a novel network GFTransformer for aerial image VQA, which could serve as a baseline for future research. The experiments show that our method achieves the best results for aerial image VQA.


\section{Introduction}
\label{sec:intro}
Since the discovery of binary black hole (BBH) merger GW150914 \citep{PhysRevLett.116.061102} by the LIGO and Virgo Scientific Collaboration, gravitational wave (GW) astronomy has gained momentum in the scientific
community by providing a new observational channel in astrophysics. This led to the first joint electromagnetic (EM) and GW multi-messenger observation of a binary neutron star (BNS) merger, GW170817 \citep{PhysRevLett.119.161101}, during the second science run of the Advanced LIGO (\cite{aLIGO} and \cite{aLIGO1}) and Advanced Virgo \citep{AdVirgo} interferometric detectors, along with a large number of BBH mergers collected in the first GW transient catalog GWTC-1 \citep{catalog}. In 2020, LIGO and Virgo completed the third science run O3; results from the first part of the run, O3a, which led to the discovery of
39 candidate binary merger events in 26 weeks of data are presented in the GWTC-2 catalogue \citep{catalog2}. Currently the LIGO, Virgo interferometers are following a schedule which will implement upgrades to increase the sensitivity for the O4 science run, to begin in 2023. The KAGRA detector \citep{KAGRA}, which started its first observing run towards the end of O3 \citep{Akutsu2020}, will also increase its sensitivity effectively producing a network of four interferometers with increased capabilities for future runs.

Among astrophysical sources of GWs, core-collapse supernovae (CCSNe) are yet to be observed and are an interesting candidate for multi-messenger analysis due to their EM and neutrino emission. However GW signals from CCSNe cannot be exactly modelled due to the stochasticity involved in the collapse dynamics and the dependency on
many parameters such as the progenitor mass, rotational state, and metallicity. Therefore alternative detection methods to the usual matched filtering approach are used which make minimal assumptions on the signal waveform, such as the wavelet-based  coherent WaveBurst pipeline (\cite{Klimenko2004} and \cite{Klimenko2016}). Machine learning (ML) techniques have been used to tackle many aspects of GW astronomy including signal detection for different types of sources (\cite{Baker2015}, \cite{Gabbard2017}, \cite{Kim2020}, and \cite{Morawski2020}), parameter estimation (\cite{Varma2019}, \cite{Haegel2020}, \cite{Green2020}, \cite{Chua2020}, and \cite{Williams2021}), noise classification (\cite{Zevin2017}, \cite{Mukund2017}, and \cite{cuoco-glitch}), and subtraction (\cite{Torres2020} and \cite{Wei2020}), as described in the comprehensive review by \cite{MLreview}. In previous work \citep{Iess2020} we showed the efficacy of 1D and 2D convolutional neural networks (CNNs) in distinguishing between noise transients, known as glitches, and waveforms from CCSNe burst signals produced through 3D hydrodynamical simulations and added to background noise, using whitened time series and spectrograms as input features. The background noise was simulated and gaussian, produced from the theoretical sensitivity curves of Virgo during O3 science run and the future Einstein Telescope (ET) detector (\cite{Punturo2010} and \cite{Hild_2011}). In the same work, we demonstrated the capability of correctly classifying among different CCSNe GW emission models. CNNs have also been used by  \cite{2018PhRvD..98l2002A} to search for g-mode emission in CCSNe signals in multiple GW detectors using a phenomenological model. \cite{Cavaglia2019} used a genetic programming algorithm which takes as input features a subset of computed parameters such as central frequency, bandwith and durations to achieve the same goal on single interferometer data. \cite{PhysRevD.102.043022} trained a 1D CNN model to search for CCSNe using the whitened GW time series from a network of GW detectors.

In this analysis we extended previous work  on the subject \citep{Iess2020} by using real detector noise from the O2 science run instead of simulated noise. We increased the number of supernova waveforms by adding two higher energy models. Moreover we explored recurrent neural networks (RNNs) and compared their performance to CNNs. RNNs are a type of neural network specialised for processing sequential data due to a feedback loop mechanism that allows the network to retain in memory previous inputs and consider these along with the current input \citep{Husken2003}. RNN models are commonly used in the fields of natural language processing \citep{Nicolaou2019} and for time-series data. In this work we used a particular type of RNN, long short-term memory cells (LSTM) \citep{Hochreiter1997}, which provide a solution to the vanishing gradient problem and have the capacity to handle long-term dependencies making them best suited for analysing gravitational wave data.
We started our analysis by training and testing on a piece of data from a single detector and later extended our approach to a three-interferometer network. The paper is structured in the following way: Section \ref{sec:dataset} and \ref{sec:waveforms} provide insight on the real detector data and a brief overview of the waveforms used in this study. Section \ref{sec:ml} describes each of the ML algorithms involved in the analysis. The results are detailed in Section \ref{sec:res}, followed by  discussions and conclusions in Section \ref{sec:disc}.

\section{Dataset}
\label{sec:dataset}
 The data analysed in this study was generated by adding selected CCSNe waveforms, obtained through 3D numerical simulations, to real interferometer noise. Compared to simulated gaussian noise used in \cite{Iess2020}, \cite{2018PhRvD..98l2002A}, and \cite{PhysRevD.102.043022}, real noise in GW detectors is subject to a certain degree of variability as it depends on environmental conditions which can be monitored but not controlled. When noise artifacts have long timescales they affect the sensitivity of the detector, which in the frequency domain corresponds to a non-stationary noise power spectral density (PSD). Moreover, many classes of short-lived non-astrophysical transients are also present in the data and can hinder GW search pipelines by triggering vetoes or false alarms. Keeping this in mind and in order to compare results, we selected 44 segments from the O2 public science run \citep{theligoscientificcollaboration2019open}, requiring that they pass the data quality flags characterised in \cite{dataquality2016} at least $97\%$ of the time for each detector. Each file contains a $4096$\,s long time series sampled at $4096$\,Hz and is identified by its initial GPS time. The segments span the period from $t_{GPS}=1185669120$ to $t_{GPS}=1186070528$. In this analysis we used five CCSN models previously tested on simulated data (s11, s13, he3.5, s18p, s25). Additionally, we included two higher amplitude models with Wolf-Rayet progenitor stars (m39, y20) and a waveform with the same progenitor mass of s18p, but no initial density perturbations in the convective oxygen shell (s18np) \citep{10.1093/mnras/staa1048}, to increase the complexity of the classification problem. The waveforms were downsampled to $4096$\,Hz, applying a low-pass filter to suppress aliasing. We simulated all signals at fixed distance of $1$\,kpc and uniformly distributed in the sky. To obtain the latter condition, we sampled from a uniform distribution in the right ascension $\alpha$ and the cosine of the declination $\delta$. The polarization angle in the wave frame was fixed at $\psi=0$. In each segment the first 300 seconds do not contain injections and were used to estimate the power spectral density (PSD) \citep{Cuoco2001} and compute the whitening parameters. The signals were then injected every ten seconds taking into account the antenna pattern functions $F_+$, $F_\times$ for the two gravitational wave polarizations $h_+$, $h_\times$ at the different detectors at each epoch, along with the time shift computed for each interferometer with respect to the geocentric frame \citep{schutz2011}. The full dependencies are described in the following formula:
\begin{equation}
    h(t) = F_+(\alpha, \delta, \lambda, \beta, \chi, \eta) \, h_+(t) + F_\times(\alpha, \delta, \lambda, \beta,\chi, \eta) \, h_\times(t),
\end{equation}
 where $\lambda$, $\beta$ are the longitude and latitude of the detector, $\chi$ defines the orientation of the bisector of the detector arms, and $\eta$ the angle between the two arms. Let $s(t) = n(t) + h(t)$ be the detector time series, with $n(t)$ interferometer noise and $h(t)$ a possible signal. The matched filter signal-to-noise ratio (S/N) time series at time $t_0$ is then computed as in  \cite{Allen2012}:
\begin{equation}
    \left(\frac{S}{N}\right)^2 = 4  \int_0^{f_{max}}{ \frac{\tilde{s}(f) \left[\tilde{h}(f)^*\right]_{t_0 = 0} }{S_n(f)} \, e^{2\pi i f t_0} \, df},
\end{equation}
where $f_{max}$ is the maximum frequency cutoff and ${S_n(f)}$ is the one-sided noise PSD. After injecting the signals into the detector strain data, a whitening step in time domain was applied by means of the Wavelet Detection Filter (WDF) library, documented in \cite{Cuoco2001} and \cite{CuocoRazzano2018}, in order to remove the stationary contribution to the noise PSD. The whitening parameters were computed by fitting the noise PSD using an Auto-Regressive (AR) model. The total number of CCSN injected in each segment is $376$, $47$ for each of the eight models. A subset of the total $16544$ CCSN signals injected in the dataset will trigger the WDF pipeline by means of a fixed threshold set on the squared sum of the wavelet transform coefficients. These will not be the only events detected by WDF, since noise transients provide additional triggers. Whitened spectrograms for instances of these two types of instrumental and astrophysical burst signals are pictured in Figure \ref{fig:samples}.

\begin{figure*}[!t]
\begin{minipage}{.6\textwidth}
\centering
    \hspace*{-1cm}
    \includegraphics[width=0.70 \textwidth]{new_glitch_misclass_L1_192.png}
\end{minipage
\begin{minipage}{.6\textwidth}
\centering
    \hspace*{-6cm}
    \includegraphics[width=0.70 \textwidth]{new_m39_sample.png}
\end{minipage}
  \caption{Sample whitened spectrograms for the noise and signal classes used in this analysis: a blip glitch (\emph{left}) and m39 (\emph{right}), a rapidly rotating CCSN explosion model. The GW shown for m39 is for emission at the pole. The spectrograms are rescaled to the $[0,1]$ range for a 2D CNN classifier.}
  \label{fig:samples}
\end{figure*}

\section{Waveforms}
\label{sec:waveforms}
\subsection{Model s11}
Model s11 is the $11\,M_{\odot}$ ZAMS model simulated by \cite{2017MNRAS.468.2032A} with the \textsc{PROMETHEUS-VERTEX} code \citep{2002A&A...396..361R}, which employs Newtonian gravity. The simulation ends 0.35\,s after the core bounce time. This model does not explode and has the smallest GW amplitude of all the GW signals considered in this study. It has a lower peak frequency at $\sim600$\,Hz.

\subsection{Model s13}
Model s13 is a $13\,M_{\odot}$ ZAMS model simulated by \cite{2019ApJ...876L...9R} using the Eulerian radiation-hydrodynamics code \textsc{FORNAX} \citep{2019ApJS..241....7S}. This model does not explode, and shows GW emission associated with g-modes. This model ends at 0.78\,s after core bounce. Due to the lack of shock revival, this model has low GW amplitude and peaks at a frequency of $\sim800$\,Hz. This model presents a weak standing accretion shock instability (SASI) component.

\subsection{Model s25}
Model s25 is a $25\,M_{\odot}$ ZAMS model simulated by \cite{2019ApJ...876L...9R} using the Eulerian radiation-hydrodynamics code \textsc{FORNAX}. It explodes at 0.5\,s after core bounce and the simulation ends at 0.62\,s after core bounce time when the GW emission is still high. This model exhibits a clear signature of the SASI at low-frequency as well as high frequency g-modes, with the peak GW emission at $\sim1000$\,Hz.

\subsection{Model s18p}
Model s18p is the $18\,M_{\odot}$ ZAMS progenitor from \cite{2018arXiv181205738P} simulated with the general relativistic neutrino hydrodynamics code \textsc{CoCoNuT-FMT}  \citep{2010ApJS..189..104M}. The simulation end time is 0.9\,s, at which time the GW emission has reached very low amplitudes. This model shows a clear g-mode signal in the spectrogram. This model explodes at $\sim300$\,ms after core bounce. The GW frequency peaks at $\sim850$\,Hz.

\subsection{Model s18np}
Model s18np from \cite{10.1093/mnras/staa1048} differs from s18 in the fact that the simulation does not include perturbations from the convective oxygen shell. As a result, this model develops strong SASI after collapse.

\subsection{Model he3.5}
Model he3.5 is the $3.5\,M_{\odot}$ ultra-stripped helium star from \cite{2018arXiv181205738P} simulated with the general relativistic neutrino hydrodynamics code \textsc{CoCoNuT-FMT}. An ultra-stripped star is a star in a binary system that has been stripped of it's outer layers due to mass transfer to the binary companion star \citep{2015MNRAS.451.2123T}. The simulation ends at 0.7\,s after core-bounce time, well after the peak GW emission phase. This model shows a clear g-mode in the spectrogram. The amplitude of the GW signal is strongest at $\sim900$\,Hz. This model explodes at $\sim0.4$\,s after core bounce.

\subsection{Model y20}
Model y20 simulated by \cite{10.1093/mnras/staa1048} with the neutrino hydrodynamics code CoCoNuT-FMT is a non-rotating, 20 solar mass Wolf-Rayet star with solar metallicity. The simulation of this model is stopped at 1.2 s after core bounce. The waveform is characterized by prompt convection after core bounce, below $200$\,Hz. Shock revival occurs $\sim200$\,ms after the bounce and is followed by a strong fundamental mode emission.

\subsection{Model m39}
Model m39 simulated by \cite{10.1093/mnras/staa1048} is a rapidly rotating Wolf-Rayet star with an initial helium star mass of 39 solar masses. The simulation is performed with the neutrino hydrodynamics code CoCoNuT-FMT. A large signal amplitude is produced by the strong neutrino-driven explosion thanks to rapid rotation ($600 \si{\km \s}^{-1}$), without the aid of strong magnetic fields. The simulation ends $0.98$\,s after core bounce and shock revival is achieved $\sim200$\,ms after bounce. The main feature is a a strong fundamental mode emission up to a peak frequency of $\sim800$\,Hz. The equatorial emission from this model has high amplitude over all frequencies associated with core bounce of a rapidly rotating star. However, we only used the polar emission in our analysis, shown in Figure \ref{fig:samples}.

\section{Deep learning algorithms}
\label{sec:ml}
For this analysis we used different types of ML algorithms for multi-label classification. Two types of inputs are provided: the whitened time series for the 1D CNN and the LSTM network, the whitened time-frequency image for the 2D CNN.

\subsection{CNN 1D}
The 1D CNN takes the whitened timeseries as inputs. It is composed of 4 convolutional layers with kernels of size 3. The number of convolutional filters are (120,80,80,40). Max pooling (2,2) and spatial dropout on $40\%$ of nodes is applied after each convolutional layer. The output from the CNN architecture is fed into two Fully Connected (FC) layers of sizes 200 and 100 respectively. The activation function used in these layers is the rectified linear unit (ReLU). The final layer is FC, with the number of nodes given by the number of class labels and a softmax activation function to yield probabilities for each class. The optimizer used is Adam, with a learning rate $\alpha = 0.001$ and categorical cross-entropy loss function. We chose a batch size of 32.
\subsection{CNN 2D}
The 2D CNN takes the whitened spectrograms as inputs. The spectrograms centered on the samples were produced with a kaiser window ($\beta=5.6$), using multiple FFTs of $0.125$\,s with overlap. The network is composed of 3 convolutional layers with kernels of sizes (4,4), (3,3), (2,2). Max pooling (2,2) was applied after each convolutional layer. The output from the CNN architecture is fed into a FC layer of size 200 and later to a FC output layer which outputs probabilities. We used the same activation functions as in the 1D case. The optimizer used is Adam, with a learning rate $\alpha = 0.001$ and categorical cross-entropy loss function. The batch size is 32.

\subsection{LSTM}
The recurrent model takes the whitened timeseries as inputs. It is composed of 2 bidirectional LSTM layers, of dimensions 64 and 32 respectively. Spatial dropout is applied on $10\%$ of the output nodes from the first recurrent layer. The outputs is passed to 4 FC layers of size 1024, 256, 64, 32 with \textit{tanh} activation functions and finally onto a FC output layer which outputs probabilities for each class through a softmax activation function.  The optimizer used is Adam, with a learning rate $\alpha = 0.001$ and categorical cross-entropy loss function. The batch size is 256.

\begin{table}[t]
    \caption{Cumulative number of triggers over all segments analysed for each interferometer individually, along with number of coincident triggers over all three detectors.}
  \begin{center}
    \begin{tabular}{|c|c|c|c|}
      \hline
      {} &

      \multicolumn{3}{c|} {\textbf{Triggers}}\\
      \hline
      \textbf{Detector} & \textit{\, Signal \, } & \textit{Noise} & \textit{\, Total \, }\\
      \hline
      Virgo V1 & $9273 $ & $47901$ & $57174$\\
      \hline
      LIGO L1 & $10480$ & $3810$ & $14290$ \\
      \hline
      LIGO H1 & $10984$ & $4103$ & $15087$ \\
      \hline
      L1, H1, V1 & $5647$ & $675$ & $6322$ \\
      \hline
    \end{tabular}
    \label{table1}
  \end{center}
\end{table}

\section{Search and classification results}
\label{sec:res}
We trained each network separately on the same training set. The full dataset was split as follows: $60\%$ training, $10\%$ validation, $30\%$ testing. In Figure \ref{fig:SNR_distribution} we show the matched filter S/N distribution of injected signals over all 44 files. The S/N of the signals varies depending on the waveform model and the simulated source position with respect to the inferferometer. As expected, the same signals produce lower S/N at Virgo due to its lower sensitivity. We observed that the matched filter S/N of lower amplitude models in Virgo amass towards an S/N equal to 4, after which the distribution drops sharply. This is consistent with the fact that we obtained the S/N as the maximum value of the matched filter response on a time series of a few seconds around each trigger. In the absence of a signal, we computed this value in the region between 3 and 4, therefore weak signals will almost never have lower S/N. Moreover, noise bursts are responsible for part of the triggers produced by WDF. In Table \ref{table1} we report the cumulative number of triggers produced over all the segments for each detector separately, along with the number of triggers coincident in the three detectors. The large excess number of noise triggers in the Virgo dataset has been obtained despite the same choice of whitening parameters when running WDF.
\begin{figure}[!h]
\centering
    \includegraphics[width=0.5\textwidth]{snr_hist_real.pdf}
  \caption{Matched filter S/N distribution of the signals injected at the ITFs: Virgo (\emph{right}) and LIGO Livingston and Hanford (\emph{left}). The LIGO detectors are grouped due to their similar sensitivity and signal S/N. A dashed line is drawn at the median S/N value of each waveform model.}
  \label{fig:SNR_distribution}
\end{figure}

This can be be explained by a non-stationary behaviour and greater variability of the detector's noise PSD  throughout the individual segments, which makes the whitening procedure less effective. This issue can be tackled in the future using adaptive whitening to track the PSD evolution in real time. For each interferometer we defined a trigger to be due to a CCSN signal when the trigger timestamp falls inside the segment covered by the injected waveform. We applied clustering when the same signal generated multiple triggers, by averaging the trigger times. Since the large number of background triggers in the V1 datum can skew the training procedure, to train this specific dataset we used only those triggers which were coincident in time with L1 and H1. The time window for coincidence was chosen to take into account the length of the waveforms and the maximum travel time between interferometers. We underline the fact that different parts of the same CCSN model can trigger WDF. Starting from the event timestamps, we chose a symmetric window around the triggers of length given by the Radice s25 model, the second shortest among the waveforms. In this way we avoided using the tails of the waveforms where the simulations are stopped due to limited computing time. We noted that the Andersen s11 waveform did not trigger the WDF pipeline a sufficient number of times in order to be effectively used during training and testing. Over the whole dataset, only 16 and 12 triggers were found coincident with s11 respectively in the LIGO H1 and LIGO L1 datasets and none coincident among all three detectors. This is the reason why we did not choose the time window for samples equal to the length of s11, while still keeping the model and its label in the training and testing procedure for completeness.
\begin{figure}[!hb]
  \centering
    \includegraphics[width=0.50\textwidth]{loss_evolutions.png}
  \caption{Evolution of the categorical cross-entropy cost function through different training epochs for the models considered. The dashed lines are training losses, while the solid
  lines are the losses computed on the validation set.}
  \label{CCE}
\end{figure}

\begin{table*}[ht!] \tiny
    \caption{True positive rates for the noise and CCSN signal classes, along with the total sensitivity, computed for the three detectors V1, L1, H1 on real data from the O2 science run. The asterisk indicates that a model is not present in the associated dataset.}
  \begin{center}
    \begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|c|c|c}
      \hline
      {} & {} & \multicolumn{10}{|c|}{\textbf{Waveform }}\\

      \hline
      {\textbf{ITF}} & \textbf{Model} & \textit{Noise} & \textit{s11} & \textit{s18p} & \textit{s18np} & \textit{He3.5} & \textit{m39} & \textit{y20} & \textit{s13} & \textit{s25} & \textbf{Total}\\
      \hline
      \multirow{3}{*}{\textit{V1}} & LSTM & $49.2$ & * & $3.6$ & $58.4$ & $0.0$ & $89.5$ & $69.9$ & $0.0$ & $89.8$ & $73.7$ \\
      & CNN 1D & $44.6$ & * & $8.4$ & $10.9$ & $0.0$ & $84.3$ & $73.1$ & $0.0$ & $87.4$ & $68.3$ \\
      & CNN 2D & $48.6$ & * & $9.6$ & $39.4$ & $3.6$ & $92.3$ & $72.5$ & $0.0$ & $94.6$ & $75.2$ \\
      \hline

      \multirow{3}{*}{\textit{L1}} & LSTM & $90.1$ & $0.0$ & $98.2$ & $92.8$ & $85.4$ & $98.7$ & $96.0$ & $87.1$ & $94.8$ & $93.6$ \\
      & CNN 1D & $99.4$ & $0.0$ & $89.5$ & $95.3$ & $82.2$ & $99.2$ & $98.2$ & $75.5$ & $98.8$ & $95.9$ \\
      & CNN 2D & $99.8$ & $0.0$ & $99.1$ & $99.3$ & $97.4$ & $100.0$ & $99.7$ & $91.6$ & $99.8$ & $99.3$\\
      \hline

      \multirow{3}{*}{\textit{H1}} & LSTM & $96.2$ & $0.0$ & $95.5$ & $96.8$ & $89.1$ & $99.7$ & $95.9$ & $75.1$ & $97.6$ & $95.4$ \\
      & CNN 1D & $99.0$ & $0.0$ & $90.1$ & $99.3$ & $91.6$ & $98.4$ & $100.0$ & $80.6$ & $97.4$ & $96.5$ \\
      & CNN 2D & $99.7$ & $0.0$ & $99.6$ & $99.8$ & $96.8$ & $99.7$ & $99.8$ & $96.8$ & $99.2$ & $99.1$ \\
      \hline
    \end{tabular}
    \label{table2}
  \end{center}
\end{table*}
Before training, the samples were shuffled to have a uniform dataset. The time needed for training varies for each model as it is dependent on the number of weights to be optimised in their architectures. Moreover, it is related to the number of samples in each dataset and is therefore shorter for Virgo due to the use of triple coincidence. The number of epochs was separately chosen for each algorithm in order to avoid overfitting. We trained the 1D CNN on a Tesla k40 GPU for 10 epochs, averaging approximately $\sim15$\,s per epoch on L1.  The 2D CNN was trained on a Tesla k40 GPU for 10 epochs, averaging approximately $\sim20$\,s per epoch on L1. The LSTM network was trained for 100 epochs on a Tesla k80 GPU, averaging approximately $\sim60$\,s per epoch on L1, as shown in Figure \ref{CCE}. In Table \ref{table2} we show the full sensitivities of the models for all the classes at each detector, which correspond to the values in the diagonal of the confusion matrices. We recall that the sensitivity, also known as the true positive rate (TPR), to a particular class $c$, is defined as the ratio of the true positives and the sum of true positives and false negatives:

\begin{equation}
    TPR_c = \frac{TP_c}{TP_c+FN_c} \, .
\end{equation}

We analyse each dataset in the following subsections, providing detailed results for the separate models and for the merged model. The predictions of the merged model were obtained by averaging the output probabilities computed by the 1D CNN, 2D CNN and LSTM. In Figure \ref{fig:multilabel_merged} we show the confusion matrices for the multi-label classification in all three interferometers, along with the S/N distribution of all the misclassified samples. In Figure \ref{misc_L1V1} we illustrate how the different CCSNe models were classified by the merged model in the LIGO and Virgo datasets as a function of S/N. The figure highlights the fact that CCSN signals have higher S/N in the LIGO datasets, with misclassifications occurring only for weaker signals. On the other hand, only the most energetic models injected into Virgo datum were correctly classified. An exception to this is provided by s25, which exhibits low-frequency SASI emission.

\subsection{LIGO L1}

After trigger extraction by WDF, the LIGO L1 dataset is composed in the following way: 3810 noise triggers, 12 s11, 1438 s18p, 1782 s18np, 704 he 3.5, 2052 m39, 1969 y20, 476 s13, 2047 s25. As previously described, the number of Andresen s11 samples is too low for any algorithm to train to recognise its characteristics. All ML models are more accurate in classifying the CCSN signals with higher energies, while the worst sensitivities are exhibited for he3.5 and s13, which are under represented in both training and test set compared to the rest. Specifically, the 1D CNN is less accurate with the s13 and s18p signals by $\sim10\%$ compared to the other algorithms. The LSTM has a small tendency to misclassify noise into low S/N signal classes, which may be due to the fact that it keeps memory of small noise peaks in the data. We merged the probabilities from the three models through averaging and obtained an overall sensitivity of $\sim99.3\%$, which is comparable to the best model (2D CNN). The incorrectly classified signals are at S/N values below 20.
\subsection{LIGO H1}
As expected the LIGO H1 dataset is quite similar to that of LIGO L1. Its composition is: 4103 noise triggers, 25 s11, 1534 s18p, 1803 s18np, 881 he3.5, 2059 m39, 1991 y20, 637 s13, 2054 s25. The high amplitude models achieve the best classification TPR for all the ML models used. As for L1, the s13 and he3.5 achieve the lowest accuracies for all ML algorithms. However, in this dataset the  LSTM architecture is the less accurate in classifying these signals. While still performing worse than the CNNs, the same model achieves better accuracy in the noise class compared to the L1 scenario. This may be due to the larger number of noise samples included.  Merging the output from the three models did not improve the sensitivity compared to the best model 2D CNN, but achieved similar values ($TPR\sim99.1\%$). As for L1, lower S/N, below 20, contribute to most of the misclassified samples.

\subsection{Virgo V1}
The triple-coincident triggers between V1, L1, and H1, which are used to identify samples in the Virgo V1 dataset, are distributed as follows: 675 noise, 0 s11, 329 s18p, 491 s18np, 115 he3.5, 1940 m39, 1139 y20, 76 s13, 1557 s25. The lower amplitude CCSN models are either not present at all (s11) or appear in a small number of samples, predominantly at lower S/N. None of the ML algorithms considered were able to effectively classify s13, he3.5, and s18p signals.\begin{figure*}[h!]
\begin{minipage}{.6\textwidth}
\centering
    \vspace*{-0.5cm}
    \includegraphics[width=0.7 \textwidth]{V1_merged_prediction.pdf}
\end{minipage
\begin{minipage}{.8\textwidth}
\centering
    \vspace*{-1.5cm}
    \hspace*{-8cm}
    \includegraphics[width=0.5 \textwidth]{misclassified_SNR_V1_merged_new.pdf}
\end{minipage}
\begin{minipage}{.6\textwidth}
\centering
\vspace*{-0.5cm}
    \includegraphics[width=0.7 \textwidth]{L1_merged_prediction.pdf}
\end{minipage
\begin{minipage}{.8\textwidth}
\centering
    \vspace*{-1.5cm}
    \hspace*{-8cm}
    \includegraphics[width=0.5 \textwidth]{misclassified_SNR_L1_merged_new.pdf}
\end{minipage}
\begin{minipage}{.6\textwidth}
\centering
    \vspace*{-0.5cm}
    \includegraphics[width=0.7 \textwidth]{H1_merged_prediction.pdf}
\end{minipage
\begin{minipage}{.8\textwidth}
\centering
    \vspace*{-1.5cm}
    \hspace*{-8cm}
    \includegraphics[width=0.5 \textwidth]{misclassified_SNR_H1_merged_new.pdf}
\end{minipage}
  \caption{Multi-label classification confusion matrices for the V1 (\emph{top}), L1 (\emph{middle}), and H1 (\emph{bottom}) datasets using the merged output from LSTM, 1D CNN, and 2D CNN, along with the S/N distribution of samples classified correctly and incorrectly. We show only the S/N range below S/N=60, where misclassifications occur.}
  \label{fig:multilabel_merged}
\end{figure*}
\begin{figure*}[h!]
\begin{minipage}{.5\textwidth}
\centering
    \hspace*{1.0cm}
    \includegraphics[width=0.65 \textwidth]{snr_misc_L1H1.pdf}
\end{minipage
\begin{minipage}{.5\textwidth}
\centering
    \hspace*{-1.5cm}
    \includegraphics[width=0.65 \textwidth]{snr_misc_V1.pdf}
\end{minipage}
  \caption{Histograms for matched filter S/N distribution of the LIGO L1, H1 (\emph{left}), and Virgo V1 (\emph{right}) test sets. The dark red thinner bars represent the number of signals misclassified by the merged LSTM and CNN model at each S/N interval. The bin widths are 2 for L1, H1 and 0.5 for V1, with the exception of m39 which has a larger S/N range. For the latter model the binning is equal to 8 for L1, H1 and 2 for V1.}
  \label{misc_L1V1}
\end{figure*} The LSTM network fares better than the CNNs for the mid-amplitude s18np model which presents a low frequency SASI emission. A possible explanation is that the LSTM is able to recognize the low frequency emission at lower S/N. The LSTM performs slightly worse than 1D CNN and 2D CNN on the y20 test samples. All models achieve good sensitivities ($TPR>84\%$) on the m39 and s25 models. The most common misidentification is with noise triggers at low S/N and with the y20 model. For V1 data the merged output improves the overall sensitivity with respect to the 2D CNN, with a $TPR\sim79.8\%$ compared to $\sim75.3\%$. As evident from Figure \ref{fig:multilabel_merged}, lower S/N<20 contribute to most of the misclassified samples, although weaker m39 signals misidentifications occur up to $S/N\sim30$. The S/N distribution of the signals exhibits a gap in the $S/N=18-30$ region: this is due to the different intrinsic GW amplitudes of the CCSN models used in the analysis and as a consequence leads to most signals being misclassified in the corresponding S/N interval.

\subsection{Three interferometer analysis}
\begin{figure}[!ht]
  \centering
    \includegraphics[width=0.5\textwidth]{all_merged_prediction_1613341014.pdf}
  \caption{Test set classification confusion matrix obtained by merging the outputs of LSTM, 1D CNN, and 2D CNN models trained on samples from a three interferometer network with LIGO (L1, H1) and Virgo (V1) data.}
  \label{3ITF}
\end{figure}

While we managed to obtain good single detector classification accuracies for the LIGO interferometers for signals at higher S/N, a more robust and realistic approach is to exploit data from all available detectors to ideally increase the accuracy for weaker signals. This can be achieved training a network on three detector data by stacking the individual time series or images, increasing the number of dimensions of the inputs by one. The CCSN signals are defined using time and equatorial coordinates in the geocentric frame. They are then projected into each interferometer datum taking into account the antenna pattern and the time shifts between the detectors. We trained and tested the models on the three interferometer samples obtained using the coincident triggers previously computed for the Virgo V1 dataset. We refer to that dataset for a breakdown of the signals used for each category. As a result the sample size is much smaller than the single LIGO detectors cases and this negatively affects the performance of the classifier. Despite such caveat, the sensitivities of the merged model to the individual CCSN models pictured in Figure \ref{3ITF}, are still very high and comparable to those obtained for the individual LIGO detectors. In detail, we see a familiar pattern where the under-represented classes and lower S/N models (he3.5, s13) have lower accuracies. On the other hand, the classification sensitivities are significantly higher than those of the Virgo dataset for all classes. As in the single detector case, the merged model is more robust in correctly classifying noise samples with a $TPR\sim97.7 \%$ in the noise class; this statement holds despite the fact that the noise class amounts to a much smaller percentage of the overall dataset, as evident in Table \ref{table1}. The overall $TPR$ for the merged model is $\sim97.8\%$. The TPR computed over all classes for the individual models confirms the single detector results and is, respectively, $TPR\sim89.5\%$ for the 1D CNN, $TPR\sim86.7\%$ for the LSTM, and $TPR\sim98.4\%$ for the 2D CNN.
Since the reduced number of triple-coincident samples hampers the training procedure, a possible solution to increase the classification accuracy is to rely on the use of generative adversarial networks (GANs) \citep{goodfellow2014generative} to augment the dataset. GANs have already been successfully applied in the generation of realistic, synthetic datasets for the variety of images of people \citep{gulrajani2017improved} or even fingertips \citep{yu2019attributing}. The same approach could in principle improve balance between various CCSN models in the studied datasets for the spectrogram representation. However, that approach would require a separate, dedicated research project.

\section{Discussion}
\label{sec:disc}
Among the sources which emit GWs that have yet to be observed, CCSNe are specifically interesting due to their multiple emission channels that convey different informations on the collapse mechanisms in play. The current matched filter techniques cannot be applied for a GW search for CCSNe, due to the large degree of stochasticity in the emitted signal. ML algorithms are largely agnostic about data and do not make assumptions on the signal morphology. However, they are well suited at detecting patterns in timeseries and time-frequency data. We have previously proven the efficacy of 1D and 2D CNNs in classifying signals based on their morphology in both domains, assuming a gaussian noise background. In this follow-up study we proved that both CNN architectures and also recurrent LSTM networks can obtain good accuracies also on classification tasks involving real detector noise. Compared to \cite{Iess2020}, this analysis presents important differences. Firstly, we added three new neutrino-driven explosion models to increase the complexity of the multi-label classification problem. Secondly, we grouped real noise transients in a single class during the training procedure. Moreover, we fixed the distance of all signals to $1$\,kpc and carried our single detector analysis on the same sources at each of the three interferometers, to compare S/N distributions and classification accuracies. As a result, the signal S/N will only depend on the sensitivities of the detectors at injection time and on the direction in the sky relative to the detector plane, through the antenna pattern functions. All three networks achieved accuracies above $90\%$ on the LIGO datasets and the predictions were consistent for the two detectors with similar sensitivity and orientation. As in the case of simulated noise background, the 2D CNN showed the best performance among the three and was very stable in its prediction for separate training runs. 1D CNN performed slightly better than LSTM for the LIGO datasets, but worse in the Virgo dataset, largely due to its inability to classify the s18np model which presents a SASI signature. The performance of the merged model is comparable to the 2D CNN, with a very high accuracy in distinguishing noise instances when fed with a large number of samples. For the Virgo dataset accuracies were expected to be worse due to the smaller sample size and the lower S/N of the CCSNe signals at the detector. Nonetheless higher amplitude signals generally achieved moderately good accuracies. On the other hand, results at S/N$<15$ for the same dataset are strongly affected by the sample size. To fully assess this low S/N region, an ad hoc dataset with a larger number of samples for each waveform can be built to probe the performance at such S/N. In this study we also extended our previous analysis \citep{Iess2020} to a multi-detector framework as in \cite{2018PhRvD..98l2002A} and \cite{PhysRevD.102.043022} by stacking the samples for a given signal from each interferometer, requiring time-coincidence in the WDF triggers. This allowed us to perform the analysis on a realistic scenario in which the LIGO datasets contain higher S/N CCSN samples compared to the Virgo interferometer. We achieved slightly lower accuracies than the single LIGO interferometer case due to the smaller dataset arising from requiring triggers to be coincident. However, we predict that increasing the number of samples in the dataset will eventually fill the gap between the two cases. From a computational point of view we are limited by dataset size and GPU memory. While we achieved similar accuracies with LSTM and 1D CNN architectures, hyper-parameter tuning was harder for the recurrent model, which also needed significantly longer training times. Moreover, LSTMs decrease their performances when faced with the task of learning longer sequences, a fact that limits their capabilities when analyzing datasets which are composed of thousands of timesteps. The better performance of LSTMs compared to 1D CNNs in the lower S/N region for some models which exhibit a low frequency emission component requires further analysis with a larger dataset. Downsampling may also be implemented to enhance the computational performance of the recurrent model and reduce the length of the sequence to be learnt. Overall, the above analysis suggests that convolutional and recurrent NN architectures can play a role in GW detection pipelines for neutrino-driven CCSN burst signals, with the added possibility of distinguishing different types of signal morphologies for sources located in a nearby region of the Milky Way. The results obtained are consistent with applications to neutrino-driven CCSN models on simulated \citep{Iess2020} and real detector noise \citep{Portillo2021}, at the fixed distance range considered in this study. Past and current estimates have set the rate of galactic CCSN between 1 and 2 events per century (\cite{1991ARA&A..29..363V}, \cite{1993A&A...273..383C}, and \cite{Rozwadowska2021}). Taking into account planned improvements in the sensitivity of the Virgo, LIGO and KAGRA interferometers, a galactic CCSN detection with deep learning methods seems possible if an event occurs with a GW signature that shares some characteristics (g-mode emission, prompt convection, SASI) with current simulations.

\section*{Acknowledgments}
\label{sec:ack}
This article/publication is based upon work from COST Action CA17137, supported by COST (European Cooperation in Science and Technology).
FM is supported by the Polish National Science Centre grants 2016/22/E/ST9/00037, 2017/26/M/ST9/00978, and 2020/37/N/ST9/02151 as well as Polish National Agency for Academic Exchange grant PPN/IWA/2019/1/00157. OL and CN are supported by the STFC UCL Centre for Doctoral Training in Data Intensive Science grant ST/P006736/1 and by an STFC consolidated grant number ST/R000476/1. Part of the computations were carried out on the PL-GRID Prometheus cluster infrastructure. This research has made use of data, software, and/or web tools obtained from the Gravitational Wave Open Science Center (https://www.gw-openscience.org/ ), a service of LIGO Laboratory, the LIGO Scientific Collaboration, and the Virgo Collaboration. LIGO Laboratory and Advanced LIGO are funded by the United States National Science Foundation (NSF) as well as the Science and Technology Facilities Council (STFC) of the United Kingdom, the Max-Planck-Society (MPS), and the State of Niedersachsen/Germany for support of the construction of Advanced LIGO and construction and operation of the GEO600 detector. Additional support for Advanced LIGO was provided by the Australian Research Council. Virgo is funded, through the European Gravitational Observatory (EGO), by the French Centre National de Recherche Scientifique (CNRS), the Italian Istituto Nazionale di Fisica Nucleare (INFN), and the Dutch Nikhef, with contributions by institutions from Belgium, Germany, Greece, Hungary, Ireland, Japan, Monaco, Poland, Portugal, and Spain. The data that support the findings of this study are available from the corresponding author upon reasonable request.

\bibliographystyle{aa}