\section{Introduction}
We present our findings from our participation in the WMT 2020 Similar Language Translation shared task, which
focused on translation between similar language
pairs in low-resource settings. Similar languages share a certain level of mutual intelligibility that may aid the improvement of translation quality. Depending on the level of closeness, certain languages may share similar orthography, lexical, syntactic, and or semantic structures which may make translation more accurate.

The level of mutual intelligibility is such that speakers of one language can understand another language without prior instruction in that other language. They can also communicate without the use of a lingua franca which is a link or vehicular language used for communicating between speakers of different languages~\citep{gooskens2007contribution}. It is important to mention that, sometimes, the level of intelligibility varies in both directions. For instance, Slovene - Croatian intelligibility is said to be asymmetric such that speakers or Slovene can understand spoken and written Croatian better than speakers of Croatian understand Slovene ~\citep{golubovic2015mutual}.

Machine translation of similar languages has been explored in a number of works~\citep{hajic2000machine, currey2016using,dabre2017empirical}. This can be seen as part of a growing need to develop models that translate well in low resource scenarios. The goal of the current shared task is to encourage researchers to explore methods for translating between similar languages. We also view the shared task as useful context for studying interaction between degrees of similarity and mutual intelligibility on the one hand, and model performance on the other hand.  We explore the use of bilingual and multilingual models for all the 5 shared task language pairs. We also perform back-translation for one language pair.

In the remainder of this paper, we discuss related literature in Section \ref{Lit-review}. We explain the methodology which includes a description of the Transformer model, back-translation and beam search in Section \ref{sec:Methodology}. In Section \ref{Models}, we describe the models we developed for this task and we discuss the various experiments we perform. We also describe the architectures of the models we developed.
Then we discuss the evaluation procedure in Section \ref{Evaluation}. Evaluation is done on both the validation and test sets. We conclude with discussion of the insights we gained from the shared task in Section \ref{Conclusion}.

\section{Related Work}\label{Lit-review}
Translation between similar languages has recently attracted attention. Different approaches have been adopted using state-of-the-art techniques, methods, and tools to take advantage of the similarity between languages even in low resource scenarios. Approaches that have been effective for other machine translation tasks have proven to achieve success in the context of similar language translation as well.

NMT models, specifically the Transformer architecture, has been shown to perform well when translating between similar languages \citep{baquero2019mllp,przystupa2019neural}. The use of in-domain data for fine-tuning has also proven to be of remarkable benefit for this task. This problem has also been tackled both by using character replacement to leverage the orthographic and phonological relationship between closely related mutually intelligible language pairs~\citep{chen2019machine}. A new approach was also introduced for this task using a two-dimensional method that assumes that each word of the target sentence can be explained by all the words in the source sentence~\citep{baquero2019mllp}.

Within the realm of MT for low resource languages, recent work has focused on translation using large monolingual corpora due to the scarcity of parallel data for many language pairs ~\citep{lample2018phrase,lample2017unsupervised,artetxe2017unsupervised}. These approaches have leveraged careful initialization of the unsupervised neural MT model using an inferred bilingual dictionary, sequence-to-sequence language models, and back-translation to achieve remarkable results. The bilingual dictionary is built without parallel data by using an unsupervised approach to align the monolingual word embedding spaces from each language ~\citep{conneau2017word,artetxe2018generalizing}. Since parallel data is not available in sufficiently large quantities, back-translation is used to create pseudo parallel data. The monolingual data of the target language is translated into the source using an existing translation system (e.g., one trained with available gold data). The output is then used to train a new MT model ~\citep{sennrich2015improving}. Weak supervision caused by back-translation results in a noisy training dataset. This eventually can affect translation quality.

More recent works adopt different approaches to manage noise in back-translation. For instance, phrase based statistical MT models are introduced as a posterior regularization during the back-translation process to reduce the noise and errors of the data generated ~\citep{ren2019unsupervised}. Another method ~\citep{artetxe2019effective} uses cross lingual word embeddings incorporated with sub-word information. The weights of the log-linear model is then tuned through an unsupervised process and the entire system is jointly refined in opposite directions to improve performance. This method out-performs previous SOTA model with about 5-7 BLUE points. A re-scoring mechanism that re-uses the pre-trained language model to select translations generated through beam search has also been found to improve fluency and consistency of translations ~\citep{liu2019incorporating}. Yet another approach, combines cross-lingual embeddings with a language model to make a phrase-table ~\citep{artetxe2019bilingual}. The resulting system is then used to generate a pseudo parallel corpus with which a bilingual lexicon is derived. This approach can work with any word or cross-lingual embeddings techniques.

\section{Methodology}\label{sec:Methodology}
Motivated by the success of Transformers and back-translation, we develop a sequence-to-sequence approach using the Transformer architecture. We also perform back-translation for one language pair. For decoding, we use Beam Search (BS). BS is an heuristic decoding strategy based on exploring the solution space and selecting a sequence of words that maximize the overall likelihood of the target sentence. During the translation, we hold a beam of $\beta$ sequences  (\textit{beam size}) which are iteratively extended. At each step, $\beta$ words are selected to extend each of the sequences in the beam, so the output is $\beta^{2}$ candidate sequences (hypotheses), we retain only the $\beta$ highest score hypotheses for the next step (top-$\beta$ candidates) \cite{koehn2009statistical}. In all our experiments we use beam size of $5$ whilst decoding.

\subsection{Transformer}
Our baseline models are based on the Transformer architecture. A Transformer  ~\citep{vaswani2017attention} is a sequence-to-sequence model that does not have the recurrent architecture present in Recurrent Neural Networks (RNNs). It uses a positional encoding that can remember how sequences are fed into the model. These positions are added to the embedded representation (n-dimensional vector) of each word. Transformers have been shown to train faster than RNNs for translation tasks.

The encoder and decoder in a Transformer model have modules that consist mainly of Multi-Head Attention multi-head attention and Feed Forward feedforward layers. The attention mechanism is based on a function that  operates on Q (\textit{queries}), K (\textit{keys}), and V (\textit{values}). The query is a vector representation of one token in the input sequence, K refers to the vector representations of all the tokens in the input sequence. More information about the Transformer are in ~\citep{vaswani2017attention}.

\subsection{Back-translation }
We perform back-translation using the monolingual model developed for the Croatian-Slovene (HR-SL) language pair. We use the best HR-SL model checkpoint that acquire the highest BLEU score on the DEV set to translate the monolingual HR data. This produces synthetic Slovene (SL) data which we then use as the source language while the original monolingual data is used as target when training the SL-HR model. We combine this data with the initial training data. Due to time constraints, we used only a subset of the monolingual data with a beam size of 5.

\subsection{Jaccard Similarity}
Jaccard similarity compares similarity, diversity, and distance of data sets \cite{niwattanakul2013using}. It is calculated between two data sets in our case two languages) A and B by dividing the number of features common to the two sets (their intersection) by the union of features in the two sets, as in (1) below:

\begin{equation}
J(A,B) = \frac{|A\cap B|}{| A \cup B|}
\end{equation}

We use tokens, identified based on white space, as features when we calculate Jaccard.

\section{Experiments} \label{Models}

\subsection{Model Architecture}
Our neural network models are based on the Transformer architecture (as described in Section \ref{sec:Methodology}) implemented by Facebook in the Fairseq toolkit. The following hyper-parameter configuration was used: $6$ attention layers in the encoder and the decoder, $4$ attention heads in each layer, embedding dimension of $512$, maximum number of tokens per batch was set to $4,096$, Adam optimizer with $\beta{1} = 0.90$, $\beta{2} = 0.98$, dropout regularization was set to $0.3$, weight-decay was set at $0.0001$, $label$-$smoothing = 0.1$, variable learning rate set at $5e^{-4}$ with the inverse square root, lr-scheduler and $warmup$-$updates = 4,000$ steps. We used the label smoothed cross-entropy criterion, and gradient clip-norm threshold was set to $0$.

\section{Data} \label{Data}
We used all the parallel data for all language pairs \url{http://www.statmt.org/wmt20/similar.html}. The task was constrained so we did not add any additional data to develop our models. We used the monolingual data for the SL-HR language pair for back-translation. Table \ref{tab:data}  shows the size of the data in terms of the number of sentences and words for each language pair while Table \ref{tab:exp} shows example source and corresponding outputs from our bilingual and multilingual models for each language pair. We also calculated the jaccard similarity for the training data we used for the tasks. ``Jaccard similarity" measures the similarity between two text documents by taking the intersection of both and dividing it by their union. Linguists measure these intersections \citep{oktavia2019understanding, gooskens2017linguistic} between languages to determine the level of mutual intelligibility as well as classify languages as dialects of the same language or different languages. We calculated Jaccard similarity for each language pair.

\begin{table*}[t]
\small
\centering
\begin{adjustbox}{width=16cm}
\renewcommand{\arraystretch}{1.5}
{
        \begin{tabular}{>{}clll}
        \toprule

      \textbf { \small Model }    &\textbf { \small Pair }   &\textbf {\small Sentence } & \textbf{ \small Translation} \\    \toprule

 \multicolumn{1}{c}{}  & \multirow{2}{*}{\textbf{Es-Ca}   } & Diseña stickers para soñar   &      \small{ Dissenya stickers per soyir }\\
 \multicolumn{1}{c}{}  &     & Mueva el diez de corazones al nueve de corazones .  &      \small{ Mobles el deu de garrons al nou de garrons . }\\
\cline{2-4}

 \multicolumn{1}{c}{}  & \multirow{2}{*}{\textbf{Es-Pt}   } & Diseña stickers para soñar     &      \small{  Design stickers para sonhar}\\
 \multicolumn{1}{c}{}  &     &Mueva el diez de corazones al nueve de corazones .  &      \small{ Muda o dez corações para nove corações .}\\
\cline{2-4}

 \multicolumn{1}{c}{}  &   \multirow{2}{*}{\textbf{Sl-Hr}   }   &Vesel pomladni pozdrav ob novi izdaji Bisnode novičk .    &      \small{ Sretan proljetni pozdrav na novom izdanju Bisnode Vijesti .}\\
 \multicolumn{1}{c}{\multirow{-4}{*}{\rotatebox[origin=c]{90}{\textbf{\small Bilingual  Model }}}}  &    & Z lepimi pozdravi ,  &      \small{  S lijepim pozdravima ,}\\

\cline{2-4}

 \multicolumn{1}{c}{}  &  \multirow{2}{*}{\textbf{Sl-Sr}   }&Danes ni enostavno slediti vsem informacijam , ki so pomembne za poslovanje .      &      \small{  Danas nije lako pratiti sve informacije koje su važne za poslovanje . }\\
 \multicolumn{1}{c}{}  &     & Iščete podatke za drugo državo ?   &      \small{ Tražite podatke za drugu zemlju ?}\\
\cline{2-4}

    \toprule

 \multicolumn{1}{c}{}  & \multirow{2}{*}{\textbf{Es-Ca}   } & Mueva el diez de corazones al nueve de corazones .     &      \small{ Muva el 10 de coração al 9 de coraons .}\\
 \multicolumn{1}{c}{}  &     &  el cuatro de diamantes        &      \small{el quatre de diamants }\\
\cline{2-4}

 \multicolumn{1}{c}{}  & \multirow{2}{*}{\textbf{Es-Pt}   } & Luche en el aire con un avión enemigo         &      \small{Luche en l aire amb un avió enemigo   }\\
 \multicolumn{1}{c}{}  &     &  Entonces , ¿ qué salió mal ?    &      \small{Então , o que saiu mal ?   }\\
\cline{2-4}

 \multicolumn{1}{c}{}  &   \multirow{2}{*}{\textbf{Sl-Hr}   }   &  Objašnjenje - Indeks plaćanja Datum :      &      \small{  Objašnjenje -- Indeks plaćanja Datum : }\\
 \multicolumn{1}{c}{\multirow{-4}{*}{\rotatebox[origin=c]{90}{\textbf{\small Multilingual  Model }}}}  &    & Poštovani ,  &      \small{Poštovani ,   }\\

\cline{2-4}

 \multicolumn{1}{c}{}  &  \multirow{2}{*}{\textbf{Sl-Sr}   }& Iščete podatke za drugo državo ?    &      \small{ Tražite podatke za drugu zemlju ?  }\\
 \multicolumn{1}{c}{}  &     & Vesel pomladni pozdrav ob novi izdaji Bisnode novičk .  &      \small{ Sretan proljetni pozdrav uz novo izdanje Bisnode novosti .  }\\
\cline{2-4}

    \toprule

\end{tabular}}
\end{adjustbox}
    \caption{Examples sentences from the various pairs and corresponding translations based on the bilingual and multilingual models. Examples are from the DEV set. }

    \label{tab:exp}
\end{table*}

\begin{table}[!ht]
\begin{center}
\textbf{}
\small
\begin{tabular}{>{}clll}
 \hline
\multicolumn{1}{c}{}  &\textbf{Language} & \textbf{\#Sentences} & \textbf{\#Words} \\
 \hline

\multicolumn{1}{c}{}  &hi & $43.2K$ & $829.9K$ \\

\multicolumn{1}{c}{}  &mr & $43.2K$ & ~$600K$  \\

\multicolumn{1}{c}{\multirow{-2}{*}{\rotatebox[origin=c]{90}{\textbf{\small Hi-Mr}}}}  &mono-hi & $113.5M$ & $4.74B$  \\

\multicolumn{1}{c}{}   &mono-mr & $4.9M$ & $112.6M$ \\

 \hlin

 \multicolumn{1}{c}{}  &es & $11.3M$ & $150.4M$ \\

\multicolumn{1}{c}{}  &ca & $11.3M$ & $163M$  \\

\multicolumn{1}{c}{\multirow{-2}{*}{\rotatebox[origin=c]{90}{\textbf{\small Es-Ca}}}}  &mono-es & $58.4M$ & $1.5B$  \\

\multicolumn{1}{c}{}   &mono-ca & $28M$ & $763.7M$ \\

 \hlin

 \multicolumn{1}{c}{}  & es & $4.15M$ & $86.6M$ \\
\multicolumn{1}{c}{}  & pt & $4.15M$  &  $82.5M$   \\

\multicolumn{1}{c}{\multirow{-2}{*}{\rotatebox[origin=c]{90}{\textbf{\small Es-pt }}}}  &mono-es &  $58.4M$   &  $1.47B$  \\

 \multicolumn{1}{c}{}  &mono-pt &  $11.4M$ & $233.9M$\\

\hlin

\multicolumn{1}{c}{}  &sl & $17.6M$ & $113.09M$ \\

\multicolumn{1}{c}{}  &hr &  $17.6M$ & $117.73M$   \\

\multicolumn{1}{c}{\multirow{-2}{*}{\rotatebox[origin=c]{90}{\textbf{\small Sl-Hr}}}}  &mono-sl & $46.25M$ & $770.6M$  \\

\multicolumn{1}{c}{}   &mono-hr & $64.5M$ & $1.24B$ \\

\hline

\multicolumn{1}{c}{}  &sl &  $14.1M$ & $79.1M$   \\

\multicolumn{1}{c}{}  &sr &  $14.1M$ & $86.1M$   \\

\multicolumn{1}{c}{\multirow{-2}{*}{\rotatebox[origin=c]{90}{\textbf{\small Sl-Sr}}}}  &mono-sl & $46.2M$ & $770.6M$  \\

\multicolumn{1}{c}{}   &mono-sr & $24M$ & $489.9M$ \\
 \toprule

\end{tabular}
\end{center}
\caption{Number of sentences and words for the training data used for each language pair.}\label{tab:data}
\end{table}

\subsection{Pre-processing}
Pre-processing was by a regular Moses toolkit ~\citep{koehn2007moses} pipeline that involved tokenization, byte pair encoding  and removing long sentences. We applied Byte-Pair Encoding (BPE) ~\citep{sennrich2015neural} operations, learned jointly over the source and target languages. For each language pair, we used $32$k split operation for subword segmentation \citep{sennrich-etal-2016-neural}. We run experiments with Transformers under three settings, as we explain next.

\subsection{Models}
We develop both bilingual and multilingual models using gold data for all pairs. For one pair, we also use back-translation with one bilingual model. We provide more details next.
\subsubsection{Bilingual Models}
In this setting, we  build an independent model for each language pair. We develop models for both directions for all language pairs, thus ultimately creating 12 models (6 for each direction). We train each model on 1 GPU for 7 days.

\subsubsection{Multilingual Models}
We develop two multilingual models that translates between all languages; a model for each direction (2 models overall) \cite{johnson2017google}. We add a language code representing the target language as the start token for each line of the source data. We train each model on 4 GPUs for 7 days. We use the same hyper-parameters values set for the bilingual models. Multilingual models enable us to determine the impact of learning similar languages with a shared representation.

\subsubsection{Bilingual Model with Back-translation }
For the third approach, we combine back-translation with the bilingual translation model for the SL-HR language pair. We incorporated the monolingual data to do this. This was influenced by report ~\citep{sennrich-etal-2016-improving} in literature on the significant improvement of translation quality when monolingual data is incorporated into training data through back-translation. We were able to test the effect of back-translation on one model.

\section{Evaluation} \label{Evaluation}
We evaluated both the DEV and TEST sets. We used the best-checkpoint metric with BLEU score to evaluate the validation set at each iteration. We used a beam size of 5 during the evaluation. We de-tokenized from BPEs back into words.

\subsection{Evaluation on DEV set}
We report the results on the DEV sets for each language pair in Table \ref{tab:bleu}. These models were trained without the monolingual data except the SL-HR pair with the asteriks.
\begin{table}[ht]
\begin{center}
\small
\begin{tabular}{p{1cm}p{2.5cm}p{2.5cm}}
 \toprule
\textbf{Pair} & \textbf{Bilingual Models} & \textbf{Multiling. models} \\
 \toprule
hi-mr & 12.14 & 16.35 \\
mr-hi & 10.63 & 01.02  \\
es-ca & 74.85 & 16.13 \\
ca-es &  74.24 & 64.57  \\
es-pt & 46.71 & 26.41 \\
pt-es & 41.12 &  05.81 \\
$*$sl-hr & 36.89 & - \\
sl-hr & 33.28 & 09.25 \\
hr-sl & 55.51 &  07.94 \\
sl-sr & 40.80 & 32.80  \\
sr-sl & 39.80 & 06.97 \\
 \toprule
\end{tabular}
\end{center}
\caption{Evaluation in BLEU on the development set for the different language pairs. The asteriks shows the model with back-translation. }
\label{tab:bleu}
\end{table}

The bilingual models outperform the multi-lingual models for all language pairs except the hi-mr language pair.
\subsection{Evaluation on TEST}
In order to evaluate the test data, we removed the byte-pair code from the test set. We used the fairseq-generate mode while translating the test set. We show results on the test set in Table \ref{tab:bleu-2}.

\begin{table}[ht]
\begin{center}
\small
\begin{tabular}{p{1cm}p{2.5cm}p{2.5cm}}
 \toprule
\textbf{Pair} & \textbf{Bilingual Models} & \textbf{Multiling. models}  \\
 \toprule
hi-mr & 0.49 & - \\

es-ca & 41.74 & 8.49 \\

ca-es &  45.23 & 45.86 \\

es-pt & 23.35 & 17.06 \\

pt-es & 24.26 &  21.55 \\

*sl-hr & 20.92 & 22.26  \\

hr-sl & 14.94 &  7.37 \\

sl-sr & 14.7 & 20.18  \\

sr-sl & 19.46 & 11.37 \\
 \toprule
\end{tabular}
\end{center}
\caption{The BLEU scores for some of the models on the test set. The language pair with the asterisks has back-translation. \footnote{We did not include results for some models because we did not receive the results before we submitted this paper.} }\label{tab:bleu-2}
\end{table}

\subsection{Discussion}
We used the Jaccard similarity to measure the level of mutual intelligbility. Figure ~\ref{fig:our_map} shows a positive correlation between the BLEU scores and the Jaccard similarity between each language pair.~\footnote{We multiplied the jaccard similarity by 100 to reduce the range of values on the y axis. } This relationship hold both for the bilingual and multilingual models. One exception is the Slovene-Serbian pair (SL-SR) where higher similarity does not translate into higher BLEU.
For example, the SL-SR BLEU is below the SL-HR BLEU even though the latter pair has a higher similarity score. Interpreting Jaccard to mean mutual intelligibility, our findings imply a higher intelligibility is correlated with higher BLEU scores. However, there is a need to further investigate this relationship due to the SL-SR we observe.

\begin{figure}[ht]
\centering
  \includegraphics[width=8cm,height=4.5cm]{jacc-sim2}
  \caption{Interaction between performance in BLEU and Jaccard similarity.}
\label{fig:our_map}
\end{figure}

\section{Conclusion} \label{Conclusion}

We described our contribution to the WMT2020  Similar Languages Translation Shared Task. We developed both bilingual and multilingual models for all pairs, in both directions. We showed back-translation to help improve performance on one pair. We also showed how mutual intelligibility between a pair of languages ( measured by Jaccard similarity) positively correlates with model performance (in BLEU). Future work can focus on exploiting other similarity metrics and providing a more in-depth study of mutual intelligibility between similar languages and how it interacts with MT model performance both in bilingual and multilingual models. The utility of back-translation on pairs we have not studied can also be fruitful.

 \section*{Acknowledgements}
 We gratefully acknowledge support from the Natural Sciences and Engineering Research Council of Canada (NSERC), the Social Sciences Research Council of Canada (SSHRC), Compute Canada (\url{www.computecanada.ca}), and UBC ARC--Sockeye (\url{https://doi.org/10.14288/SOCKEYE}).


\section{Introduction}

Encouraged by the growing availability of low-cost sensors, \emph{multimodal fusion} that takes advantage of data obtained from different sources/structures for classification or regression has become a central problem in machine learning~\cite{journals/pami/BaltrusaitisAM19}. Joining the success of deep learning, multimodal fusion is recently specified as \emph{deep multimodal fusion} by introducing end-to-end neural integration of multiple modalities~\cite{ramachandram2017deep}, and it has exhibited remarkable benefits
against the unimodal paradigm in semantic segmentation~\cite{conf/iccv/LeePH17,journals/ijcv/ValadaMB20}, action recognition~\cite{fan2018end,conf/eccv/GarciaMM18,journals/tip/SongLLG20}, visual question answering~\cite{conf/iccv/AntolALMBZP15,conf/nips/IlievskiF17}, and many others~\cite{conf/iccv/BalntasDSSKK17,conf/iclr/JinYBJ19,conf/iccv/ZhangZSWSL19}.

A variety of works have been done towards deep multimodal fusion~\cite{ramachandram2017deep}. Regarding the type of how they fuse, existing methods are generally categorized into \emph{aggregation-based} fusion, \emph{alignment-based} fusion, and the mixture of them~\cite{journals/pami/BaltrusaitisAM19}. The aggregation-based methods employ a certain operation (\emph{e.g.} averaging \cite{conf/accv/HazirbasMDC16}, concatenation~\cite{conf/icml/NgiamKKNLN11,conf/cvpr/ZengTHYSCW19}, and self-attention~\cite{journals/ijcv/ValadaMB20}) to combine multimodal sub-networks into a single network. The alignment-based fusion~\cite{conf/cvpr/ChengCLZH17,journals/tip/SongLLG20,conf/eccv/WangWTSW16}, instead, adopts a regulation loss to align the embedding of all sub-networks while keeping full propagation for each of them. The difference between such two mechanisms is depicted in Figure~\ref{pic:sketch}. Another categorization of multimodal fusion can be specified as early, middle, and late fusion, depending on when to fuse, which have been discussed in earlier works~\cite{atrey2010multimodal,bruni2014multimodal,hall1997introduction,snoek2005early} and also in the deep learning literature~\cite{journals/pami/BaltrusaitisAM19,kiela2017deep,lazaridou2014wampimuk,conf/nips/VriesSMLPC17}.

Albeit the fruitful progress, it remains a great challenge on how to integrate the common information across modalities, meanwhile preserving the specific patterns of each one. In particular, the aggregation-based fusion is prone to underestimating the intra-modal propagation once the multimodal sub-networks have been aggregated. On the contrary, the alignment-based fusion maintains the intra-modal propagation, but it always delivers ineffective inter-modal fusion owing to the weak message exchanging by solely training the alignment loss. To balance between inter-modal fusion and intra-modal processing, current methods usually resort to careful hierarchical combination of the aggregation and alignment fusion for enhanced performance, at a cost of extra computation and engineering overhead~\cite{conf/cvpr/DuWWZW19,conf/iccv/LeePH17,conf/cvpr/ZengTHYSCW19}.

\textbf{Present Work.}
We propose Channel-Exchanging-Network (CEN) which is parameter-free, adaptive, and effective. Instead of using aggregation or alignment as before, CEN dynamically exchanges the channels between sub-networks for fusion (see Figure~\ref{pic:sketch}(c)). The core of CEN lies in its smaller-norm-less-informative assumption inspired from network pruning~\cite{conf/iccv/LiuLSHYZ17,conf/iclr/YeL0W18}. To be specific, we utilize the scaling factor (\emph{i.e.} $\gamma$) of Batch-Normalization (BN)~\cite{conf/icml/IoffeS15} as the importance measurement of each corresponding channel, and replace the channels associated with close-to-zero factors of each modality with the mean of other modalities. Such message exchanging is parameter-free and self-adaptive, as it is dynamically controlled by the scaling factors that are determined by the training itself.
Besides, we only allow directed channel exchanging within a certain range of channels in each modality to preserve intra-modal processing. More details are provided in \textsection~\ref{subsec:channel_exc}.
Necessary theories on the validity of our idea are also presented in \textsection~\ref{subsec_anslysis}.

Another hallmark of CEN is that the parameters except BN layers of all sub-networks are shared with each other (\textsection~\ref{subsec:net_sharing}). Although this idea is previously studied in~\cite{conf/cvpr/ChangYSKH19,wang2020asymfusion}, we apply it here to serve specific purposes in CEN: by using private BNs, as already discussed above, we can determine the channel importance for each individual modality;
by sharing convolutional filters, the corresponding channels among different modalities are embedded with the same mapping, thus more capable of modeling the modality-common statistic. This design further compacts the multimodal architecture to be almost as small as the unimodal one.

We evaluate our CEN on two studies: semantic segmentation via RGB-D data~\cite{conf/eccv/SilbermanHKF12,conf/cvpr/SongLX15} and image translation through multi-domain input~\cite{conf/cvpr/ZamirSSGMS18}.
It demonstrates that CEN yields remarkably superior performance than various kinds of fusion methods based on aggregation or alignment under a fair condition of comparison. In terms of semantic segmentation particularly, our CEN significantly outperforms state-of-the-art methods on two popular benchmarks. We also conduct ablation studies to isolate the benefit of each proposed component. More specifications are provided in \textsection~\ref{sec:experiments}.

\begin{figure}[t]
\centering
\includegraphics[scale=0.27]{pics/fusion.pdf}
\caption{A sketched comparison between existing fusion methods and ours.}
\label{pic:sketch}
\end{figure}

\section{Related Work}
We introduce the methods of deep multimodal fusion, and the concepts related to our paper.

\textbf{Deep multimodal fusion.}
As discussed in introduction, deep multimodal fusion methods can be mainly categorized into aggregation-based fusion and alignment-based fusion~\cite{journals/pami/BaltrusaitisAM19}.
Due to the weakness in intra-modal processing, recent aggregation-based works perform feature fusion while still maintaining the sub-networks of all modalities~\cite{conf/cvpr/DuWWZW19,conf/iccv/LinCCHH17}.
Besides, \cite{conf/accv/HazirbasMDC16} points out the performance by fusion is highly affected by the choice of which layer to fuse. Alignment-based fusion methods align multimodal features by applying the similarity regulation, where Maximum-Mean-Discrepancy (MMD)~\cite{journals/jmlr/GrettonBRSS12} is usually adopted for the measurement.
However, simply focusing on unifying the whole distribution may overlook the specific patterns in each domain/modality \cite{conf/nips/BousmalisTSKE16,journals/tip/SongLLG20}. Hence, \cite{conf/eccv/WangWTSW16} provides a way that may alleviate this issue, which correlates modality-common features while simultaneously maintaining modality-specific information.
There is also a portion of the multimodal learning literature based on modulation~\cite{de2017guesswhat,dumoulin2018feature,conf/nips/VriesSMLPC17}. Different from these types of fusion methods, we propose a new fusion method by channel exchanging, which potentially enjoys the guarantee to both sufficient inter-model interactions and intra-modal learning.

\textbf{Other related concepts.}
The idea of using BN scaling factor to evaluate the importance of CNN channels has been studied in network pruning~\cite{conf/iccv/LiuLSHYZ17,conf/iclr/YeL0W18} and representation learning~\cite{shao2020channel}.
Moreover, \cite{conf/iccv/LiuLSHYZ17} enforces $\ell_1$ norm penalty on the scaling factors and explicitly prunes out filters meeting a sparsity criteria. Here, we apply this idea as an adaptive tool to determine where to exchange and fuse.  CBN~\cite{conf/nips/VriesSMLPC17} performs cross-modal message passing by modulating BN of one modality conditional on the other, which is clearly different from our method that directly exchanges channels between different modalities for fusion.
ShuffleNet~\cite{conf/cvpr/ZhangZLS18} proposes to shuffle a portion of channels among multiple groups for efficient propagation in light-weight networks, which is similar to our idea of exchanging channels for message fusion. Yet, while the motivation of our paper is highly different, the exchanging process is self-determined by the BN scaling factors, instead of the random exchanging in ShuffleNet.

\section{Channel Exchanging Networks}
In this section, we introduce our CEN, by mainly specifying its two fundamental components: the channel exchanging process and the sub-network sharing mechanism, followed by necessary analyses.

\subsection{Problem Definition}
Suppose we have the $i$-th input data of $M$ modalities, $\bm{x}^{(i)}=\{\bm{x}_m^{(i)}\in\mathbb{R}^{C\times (H\times W)}\}_{m=1}^M$, where $C$ denotes the number of channels, $H$ and $W$ denote the height and width of the feature map\footnote{Although our paper is specifically interested in image data, our method is still general to other domains; for example, we can set $H=W=1$ for vectors.}. We define $N$ as the batch-size.
The goal of deep multimodal fusion is to determine a multi-layer network $f(\bm{x}^{(i)})$ (particularly CNN in this paper) whose output $\hat{\bm{y}}^{(i)}$ is expected to fit the target $\bm{y}^{(i)}$ as much as possible. This can be implemented by minimizing the empirical loss as
\begin{equation}
\label{eq:loss}
    \min_{f}\frac{1}{N}\sum_{i=1}^{N}\mathcal{L}\left(\hat{\bm{y}}^{(i)}=f(\bm{x}^{(i)}),\bm{y}^{(i)}\right).
\end{equation}

We now introduce two typical kinds of instantiations to Equation~\ref{eq:loss}:

\textbf{I.}
The aggregation-based fusion first processes each $m$-th modality with a separate sub-network $f_m$ and then combine all their outputs via an aggregation operation followed by a global mapping. In formal, it computes the output by
\begin{equation}
    \hat{\bm{y}}^{(i)}=f(\bm{x}^{(i)})=h(\text{Agg}(f_1(\bm{x}_1^{(i)}),\cdots,f_M(\bm{x}_M^{(i)}))),
\end{equation}
where $h$ is the global network and $\text{Agg}$ is the aggregation function. The aggregation can be implemented as averaging~\cite{conf/accv/HazirbasMDC16}, concatenation~\cite{conf/cvpr/ZengTHYSCW19}, and self-attention~\cite{journals/ijcv/ValadaMB20}. All networks are optimized via minimizing Equation~\ref{eq:loss}.

\textbf{II.}
The alignment-based fusion leverages an alignment loss for capturing the inter-modal concordance while keeping the outputs of all sub-networks $f_m$. Formally, it solves
\begin{equation}
\label{eq:alig}
    \min_{f_{1:M}} \frac{1}{N}\sum_{i=1}^{N} \mathcal{L}\left(\sum_{m=1}^M\alpha_mf_m(\bm{x}_m^{(i)}), \bm{y}^{(i)}\right)+ \text{Alig}_{f_{1:M}}(\bm{x}^{(i)}), \quad s.t. \sum_{m=1}^M\alpha_m=1,
\end{equation}
where the alignment $\text{Alig}_{f_{1:M}}$ is usually specified as Maximum-Mean-Discrepancy (MMD)~\cite{journals/jmlr/GrettonBRSS12} between certain hidden features of sub-networks, and the final output $\sum_{m=1}^M\alpha_mf_m(\bm{x}_m^{(i)})$ is an ensemble of $f_m$ associated with the decision score $\alpha_m$ which is learnt by an additional softmax output to meet the simplex constraint.

As already discussed in introduction, both fusion methods are insufficient to determine the trade-off between fusing modality-common information and preserving modality-specific patterns. In contrast, our CEN is able to combine their best, the details of which are clarified in the next sub-section.

\subsection{Overall Framework}
The whole optimization objective of our method is
\begin{equation}
\label{eq:cen}
    \min_{f_{1:M}} \frac{1}{N}\sum_{i=1}^{N} \mathcal{L}\left(\sum_{m=1}^M\alpha_mf_m(\bm{x}^{(i)}), \bm{y}^{(i)}\right)+ \lambda\sum_{m=1}^{M}\sum_{l=1}^L|\bm{\hat{\gamma}}_{m,l}|, \quad s.t. \sum_{m=1}^M\alpha_m=1,
\end{equation}
where,
\vspace{-0.3em}
\begin{itemize}
    \item The sub-network $f_m(\bm{x}^{(i)})$ (opposed to $f_m(\bm{x}_m^{(i)})$ in Equation~\ref{eq:alig} of the alignment fusion) fuses multimodal information by channel exchanging, as we will detail in \textsection~\ref{subsec:channel_exc};
    \item Each sub-network is equipped with BN layers containing the scaling factors $\bm{\gamma}_{m,l}$ for the $l$-th layer, and we will penalize the
    $\ell_1$ norm of their certain portion $\bm{\hat{\gamma}}_{m,l}$ for sparsity, which is presented in \textsection~\ref{subsec:channel_exc};
    \item The sub-network $f_m$ shares the same parameters except BN layers to facilitate the channel exchanging as well as to compact the architecture further, as introduced in \textsection~\ref{subsec:net_sharing};
    \item The decision scores of the ensemble output, $\alpha_m$, are trained by a softmax output similar to the alignment-based methods.
\end{itemize}

By the design of Equation~\ref{eq:cen}, we conduct a parameter-free message fusion across modalities while maintaining the self-propagation of each sub-network so as to characterize the specific statistic of each modality. Moreover, our fusion of channel exchanging is self-adaptive and easily embedded to everywhere of the sub-networks, with the details given in what follows.

\begin{figure}[t]
\centering
\includegraphics[scale=0.3]{pics/channel.pdf}
\vskip -0.05 in
\caption{An illustration of our multimodal fusion strategy. The sparsity constraints on scaling factors are applied to disjoint regions of different modalities. A feature map will be replaced by that of other modalities at the same position, if its scaling factor is lower than a threshold. }
\label{channel}
\vskip -0.1 in
\end{figure}

\subsection{Channel Exchanging by Comparing BN Scaling Factor}
\label{subsec:channel_exc}
Prior to introducing the channel exchanging process, we first review the BN layer~\cite{conf/icml/IoffeS15}, which is used widely in deep learning to eliminate covariate shift and improve generalization. We denote by $\bm{x}_{m,l}$ the $l$-th layer feature maps of the $m$-th sub-network, and by $\bm{x}_{m,l,c}$ the $c$-th channel. The BN layer performs a  normalization of $\bm{x}_{m,l}$ followed by an affine transformation, namely,
\begin{equation}
\label{eq:bn}
\bm{x}'_{m,l,c} = \gamma_{m,l,c}\frac{\bm{x}_{m,l,c}-\mu_{m,l,c}}{\sqrt{\sigma^2_{m,l,c}+\epsilon}}+\beta_{m,l,c},
\end{equation}
where, $\mu_{m,l,c}$ and $\sigma_{m,l,c}$ compute the mean and the standard deviation, respectively, of all activations over all pixel locations ($H$ and $W$) for the current mini-batch data; $\gamma_{m,l,c}$ and $\beta_{m,l,c}$ are the trainable scaling factor and offset, respectively; $\epsilon$ is a small constant to avoid divisions by zero. The $(l+1)$-th layer takes $\{\bm{x}'_{m,l,c}\}_c$ as input after a non-linear function.

The factor $\gamma_{m,l,c}$ in Equation~\ref{eq:bn} evaluates the correlation between the input $\bm{x}_{m,l,c}$ and the output $\bm{x}'_{m,l,c}$ during training. The gradient of the loss \emph{w.r.t.} $\bm{x}_{m,l,c}$ will approach 0 if $\gamma_{m,l,c}\rightarrow 0$, implying that $\bm{x}_{m,l,c}$ will lose its influence to the final prediction and become redundant thereby. Moreover, we will prove in \textsection~\ref{subsec_anslysis} that the state of $\gamma_{m,l,c}=0$ is attractive with a high probability, given the $\ell_1$ norm regulation in Equation~\ref{eq:cen}. In other words, once the current channel $\bm{x}_{m,l,c}$ becomes redundant due to $\gamma_{m,l,c}\rightarrow 0$ at a certain training step, it will almost do henceforth.

It thus motivates us to replace the channels of small scaling factors with the ones of other sub-networks, since those channels potentially are redundant. To do so, we derive
\begin{equation}
\label{eq:exchange-bn}
\bm{x}'_{m,l,c} =
\begin{cases}
\gamma_{m,l,c}\frac{\bm{x}_{m,l,c}-\mu_{m,l,c}}{\sqrt{\sigma^2_{m,l,c}+\epsilon}}+\beta_{m,l,c}, &\text{if}\quad \gamma_{m,l,c}>\theta; \\
\frac{1}{M-1}\sum\limits_{m'\neq m}^M\gamma_{m',l,c}\frac{\bm{x}_{m',l,c}-\mu_{m',l,c}}{\sqrt{\sigma^2_{m',l,c}+\epsilon}}+\beta_{m',l,c}, &\text{else};
\end{cases}
\end{equation}
where, the current channel is replaced with the mean of other channels if its scaling factor is smaller than a certain threshold $\theta\approx0^+$. In a nutshell, if one channel of one modality has little impact to the final prediction, then we replace it with the mean of other modalities. We apply Equation~\ref{eq:exchange-bn} for each modality before feeding them into the nonlinear activation followed by the convolutions in the next layer. Gradients are detached from the replaced channel and back-propagated through the new ones.

In our implementation, we divide the whole channels into $M$ equal sub-parts, and only perform the channel exchanging in each different sub-part for different modality. We denote the scaling factors that are allowed to be replaced as $\bm{\hat{\gamma}}_{m,l}$. We further impose the sparsity constraint on $\bm{\hat{\gamma}}_{m,l}$ in Equation~\ref{eq:cen} to discover unnecessary channels. As the exchanging in Equation~\ref{eq:exchange-bn} is a directed process within only one sub-part of channels, it hopefully can not only retain modal-specific propagation in the other $M-1$ sub-parts but also avoid unavailing exchanging since $\gamma_{m',l,c}$, different from $\hat{\gamma}_{m,l,c}$, is out of the sparsity constraint. Figure~\ref{channel} illustrates our channel exchanging process.

\subsection{Sub-Network Sharing with Independent BN}
\label{subsec:net_sharing}
It is known in~\cite{conf/cvpr/ChangYSKH19,wang2020asymfusion} that leveraging private BN layers is able to characterize the traits of different domains or modalities. In our method, specifically, different scaling factors (Equation~\ref{eq:bn}) evaluate the importance of the channels of different modalities, and they should be decoupled.

With the exception of BN layers, all sub-networks $f_m$ share all parameters with each other including convolutional filters\footnote{If the input channels of different modalities are different (\emph{e.g.} RGB and depth), we will broaden their sizes to be the same as their Least Common Multiple (LCM).}. The hope is that we can further reduce the network complexity and therefore improve the predictive generalization. Rather, considering the specific design of our framework, sharing convolutional filters is able to capture the common patterns in different modalities, which is a crucial purpose of multimodal fusion. In our experiments, we conduct multimodal fusion on RGB-D images or on other domains of images corresponding to the same image content. In this scenario, all modalities are homogeneous in the sense that they are just different views of the same input. Thus, sharing parameters between different sub-networks still yields promisingly expressive power. Nevertheless, when we are dealing with heterogeneous modalities (\emph{e.g.} images with text sequences), it would impede the expressive power of the sub-networks if keeping sharing their parameters, hence a more dexterous mechanism is suggested, the discussion of which is left for future exploration.

\subsection{Analysis}
\label{subsec_anslysis}
\begin{theorem}
\label{th:theorem}
Suppose $\{\gamma_{m,l,c}\}_{m,l,c}$ are the BN scaling factors of any multimodal fusion network (without channel exchanging) optimized by Equation~\ref{eq:cen}. Then the probability of $\gamma_{m,l,c}$ being attracted to $\gamma_{m,l,c}=0$  during training (\emph{a.k.a.} $\gamma_{m,l,c}=0$ is the local minimum) is equal to $2\Phi(\lambda|\frac{\partial L}{\partial \bm{x}'_{m,l,c}}|^{-1})-1$, where $\Phi$ derives the cumulative probability of standard Gaussian.
\end{theorem}

In practice, especially when approaching the convergence point, the magnitude of $\frac{\partial L}{\partial \bm{x}'_{m,l,c}}$ is usually very close to zero, indicating that the probability of staying around $\gamma_{m,l,c}=0$ is large. In other words, when the scaling factor of one channel is equal to zero, this channel will almost become redundant during later training process, which will be verified by our experiment in the appendix. Therefore, replacing the channels of $\gamma_{m,l,c}=0$ with other channels (or anything else) will only enhance the trainablity of the model. We immediately have the following corollary,
\setcounter{theorem}{0}
\begin{corollary}
\label{th:corollary}
\vspace{-0.1em}
If the minimal of Equation~\ref{eq:cen} implies $\gamma_{m,l,c}=0$, then the channel exchanging by Equation~\ref{eq:exchange-bn} (assumed no crossmodal parameter sharing) will only decrease the training loss, \emph{i.e.} $\min_{f'_{1:M}}L\leq\min_{f_{1:M}}L$, given the sufficiently expressive $f'_{1:M}$ and $f_{1:M}$ which denote the cases with and without channel exchanging, respectively.
\end{corollary}

\section{Experiments}
\label{sec:experiments}
We contrast the performance of CEN against existing multimodal fusion methods on two different tasks: semantic segmentation and image-to-image translation. The frameworks for both tasks are in the encoder-decoder style. Note that we only perform multimodal fusion within the encoders of different modalities throughout the experiments. Our codes are complied on PyTorch~\cite{conf/nips/PaszkeGMLBCKLGA19}.

\input{tabs/slim_ablation.tex}
\input{tabs/refinenet.tex}
\input{tabs/sunrgbd_nyudv2.tex}

\subsection{Semantic Segmentation}
\textbf{Datasets.}
We evaluate our method on two public datasets NYUDv2~\cite{conf/eccv/SilbermanHKF12} and SUN RGB-D~\cite{conf/cvpr/SongLX15}, which consider RGB and depth as input. Regarding NYUDv2, we follow the standard settings and adopt the split of 795 images for training and 654 for testing, with predicting standard 40 classes~\cite{conf/cvpr/GuptaAM13}. SUN RGB-D is one of the most challenging large-scale benchmarks towards indoor semantic segmentation, containing 10,335 RGB-D images of 37 semantic classes. We use the public train-test split (5,285 vs 5,050).

\textbf{Implementation.} We consider RefineNet~\cite{lin2019refinenet}/PSPNet~\cite{conf/cvpr/ZhaoSQWJ17} as our segmentation framework whose backbone is implemented by ResNet~\cite{conf/cvpr/HeZRS16} pretrained from ImageNet dataset \cite{journals/ijcv/RussakovskyDSKS15}.
The initial learning rates are set to $5\times10^{-4}$ and $3\times10^{-3}$ for the encoder and decoder, respectively, both of which are reduced to their halves every 100/150 epochs (total epochs 300/450) on NYUDv2 with ResNet101/ResNet152 and every 20 epochs (total epochs 60) on SUN RGB-D. The mini-batch size, momentum and weight decay are selected as 6, 0.9, and $10^{-5}$, respectively, on both datasets.
We set $\lambda=5\times10^{-3}$ in Equation~\ref{eq:cen} and the threshold to $\theta=2\times10^{-2}$ in Equation~\ref{eq:exchange-bn}.
Unless otherwise specified, we adopt the multi-scale strategy~\cite{conf/iccv/LeePH17,lin2019refinenet} for test.
We employ the Mean IoU along with Pixel Accuracy and Mean Accuracy as evaluation metrics following~\cite{lin2019refinenet}. Full implementation details are referred to our appendix.

\textbf{The validity of each proposed component.}
Table \ref{tabs:component} summarizes the results of different variants of CEN on NYUDv2. We have the following observations:
\textbf{1.} Compared to the unshared baseline, sharing the convolutional parameters greatly boosts the performance particularly on the Depth modality (35.8 vs 38.4). Yet, the performance will encounter a clear drop if we additionally share the BN layers. This observation is consistent with our analyses in~\textsection~\ref{subsec:net_sharing} due to the different roles of convolutional filters and BN parameters.
\textbf{2.} After carrying out directed channel exchanging under the $\ell_1$ regulation, our model gains a huge improvement on both modalities, \emph{i.e.} from 46.0 to 49.7 on RGB, and from 38.1 to 45.1 on Depth, and finally increases the ensemble Mean IoU from 47.6 to 51.1. It thus verifies the effectiveness of our proposed mechanism on this task.
\textbf{3.}
Note that the channel exchanging is only available on a certain portion of each layer (\emph{i.e.} the half of the channels in the two-modal case). When we remove this constraint and allow all channels to be exchanged by Equation~\ref{eq:exchange-bn}, the accuracy decreases, which we conjecture is owing to the detriment by impeding modal-specific propagation, if all channels are engaged in cross-modal fusion.

To further explain why channel exchanging works, Figure \ref{featuremap5} displays the feature maps of RGB and Depth, where we find that the RGB channel with non-zero scaling factor mainly characterizes the texture, while the Depth channel with non-zero factor focuses more on the boundary; in this sense, performing channel exchanging can better combine the complementary properties of both modalities.

\textbf{Comparison with other fusion baselines.}
Table~\ref{tabs:our_implementation} reports the comparison of our CEN with two aggregation-based methods: concatenation~\cite{conf/cvpr/ZengTHYSCW19} and self-attention~\cite{journals/ijcv/ValadaMB20}, and one alignment-based approach~\cite{conf/eccv/WangWTSW16},  using the same backbone. All baselines are implemented with the early, middle, late, and all stage fusion. Besides, for a more fair comparison, all baselines are further  conducted under the same setting (except channel exchanging) with ours, namely, sharing convolutions with private BNs, and preserving the propagation of all sub-networks. Full details are provided in the appendix. It demonstrates that, on both settings, our method always outperforms others by an average improvement more than 2\%. We also report the parameters used for fusion, \emph{e.g.} the aggregation weights of two modalities in concatenation. While self-attention (all-stage) attains the closest performance to us (49.1 vs 51.1), the parameters it used for fusion are considerable, whereas our fusion is parameter-free.

\textbf{Comparison with SOTAs.}
We contrast our method against a wide range of state-of-the-art methods. Their results are directly copied from previous papers if provided or re-implemented by us otherwise, with full specifications illustrated in the appendix.
Table~\ref{tabs:seg_results} concludes that our method equipped with PSPNet (ResNet152) achieves new records remarkably superior to previous methods in terms of all metrics on both datasets. In particular, given the same backbone, our method are still much better than RDFNet~\cite{conf/iccv/LeePH17}. To isolate the contribution of RefineNet in our method, Table~\ref{tabs:seg_results} also provides the uni-modal results, where we observe a clear advantage of multimodal fusion.

\textbf{Additional ablation studies.}
In this part, we provide some additional experiments on NYUDv2, with RefineNet (ResNet101). Results are obtained with single-scale evaluation.
\textbf{1.} As $\ell_1$ enables the discovery of unnecessary channels and comes as a pre-condition of Theorem~\ref{th:theorem}, naively exchanging channels with a fixed portion (without using $\ell_1$ and threshold) could not reach good performance. For example, exchanging a fixed portion of 30\% channels only gets IoU 47.2. We also find by only exchanging 30\% channels at each down-sampling stage of the encoder, instead of every $3\times3$ convolutional layer throughout the encoder (like our CEN), the result becomes 48.6, which is much lower than our CEN (51.1). \textbf{2.} In Table~\ref{tabs:seg_results}, we provide results of our implemented CBN~\cite{conf/nips/VriesSMLPC17} by modulating the BN of depth conditional on RGB. The IoUs of CBN with unshared and shared convolutional parameters are 48.3 and 48.9, respectively.
\textbf{3.} Directly summing activations (discarding the 1st term in Equation~\ref{eq:exchange-bn}) results in IoU 48.1, which could reach 48.4 when summing with a learnt soft gate. \textbf{4.} If we replace the ensemble of expert with a concat-fusion block, the result will slightly reduce from 51.1 to 50.8. \textbf{5.} Besides, we try to exchange channels randomly like ShuffleNet or directly discard unimportant channels without channel exchanging, the IoUs of which are 46.8 and 47.5, respectively. All above ablations support the optimal design of our architecture.

\begin{figure}[t]
\centering
\hskip -0.1 in
\includegraphics[scale=0.22]{pics/featuremap.pdf}
\vskip -0.1 in
\caption{ Visualization of the averaged feature maps for RGB and Depth. From left to right: the input images, the channels of $(\gamma_{rgb}\approx 0, \gamma_{depth}> 0)$, $(\gamma_{rgb}> 0, \gamma_{depth}\approx 0)$, and $(\gamma_{rgb}> 0, \gamma_{depth}> 0)$.}
\label{featuremap5}

\vskip -0.06 in
\end{figure}

\input{tabs/taskonomy.tex}
\input{tabs/multimodal.tex}

\subsection{Image-to-Image Translation}
\textbf{Datasets.} We adopt Taskonomy~\cite{conf/cvpr/ZamirSSGMS18}, a dataset with 4 million images of indoor scenes of about 600 buildings. Each image in Taskonomy has more than 10 multimodal representations, including depth (euclidean/zbuffer), shade, normal, texture, edge, principal curvature, etc. For efficiency, we sample 1,000 high-quality multimodal images for training, and 500 for validation.

\textbf{Implementation.} Following Pix2pix~\cite{conf/cvpr/IsolaZZE17}, we adopt the U-Net-256 structure for image translation with the consistent setups with~\cite{conf/cvpr/IsolaZZE17}. The BN computations are replaced with Instance Normalization layers (INs), and our method (Equation~\ref{eq:exchange-bn}) is still applicable. We adopt individual INs in the encoder, and share all other parameters including INs in the decoder. We set $\lambda$ to $10^{-3}$ for sparsity constraints and the threshold $\theta$ to $10^{-2}$. We adopt FID~\cite{conf/nips/HeuselRUNH17} and KID~\cite{conf/iclr/BinkowskiSAG18} as evaluation metrics, which will be introduced in our appendix.

\textbf{Comparison with other fusion baselines.}
In Table~\ref{tabs:translation}, we evaluate the performance on two specific translation cases, \emph{i.e.} Shade+Texture$\to$RGB and Depth+Normal$\to$RGB, with more examples included in the appendix. In addition to the three baselines used in semantic segmentation (Concat, Self-attention, Align),  we conduct an extra aggregation-based method by using the average operation. All baselines perform fusion under 4 different kinds of strategies: early (at the 1st conv-layer), middle (the 4th conv-layer), late (the 8th conv-layer), and all-layer fusion. As shown in Table~\ref{tabs:translation}, our method yields much lower FID/KID than others, which supports the benefit of our proposed idea once again.

\textbf{Considering more modalities.}
We now test whether our method is applicable to the case with more than 2 modalities. For this purpose, Table~\ref{tabs:multimodal} presents the results of image translation to RGB by inputting from 1 to 4 modalities of Depth, Normal, Texture, and Shade. It is observed that increasing the number of modalities improves the performance consistently, suggesting much potential of applying our method towards various cases.

\section{Conclusion}
\vskip -0.03 in
In this work, we propose Channel-Exchanging-Network (CEN), a novel framework for deep multimodal fusion, which differs greatly with existing aggregation-based and alignment-based multimodal fusion. The motivation behind is to boost inter-modal fusion while simultaneously keeping sufficient intra-modal processing. The channel exchanging is self-guided by channel importance measured by individual BNs, making our framework self-adaptive and compact. Extensive evaluations verify the effectiveness of our method.

\section*{Acknowledgement}
This work is jointly funded by National Natural Science Foundation of China and German Research Foundation (NSFC 61621136008/DFG TRR-169) in project ``Crossmodal Learning'' \textrm{II}, Tencent AI Lab Rhino-Bird Visiting Scholars Program (VS202006), and China Postdoctoral
Science Foundation (Grant No.2020M670337).

\section*{Broader Impact}
This research enables fusing complementary information from different modalities effectively, which helps improve performance for autonomous vehicles and indoor manipulation robots, also making them more robust to environmental conditions, \emph{e.g.} light, weather. Besides, instead of carefully designing hierarchical fusion strategies in existing methods, a global criterion is applied in our work for guiding multimodal fusion, which allows easier model deployment for practical applications. A drawback of bringing deep neural networks into multimodal fusion is its insufficient interpretability.

{\small
\bibliographystyle{splncs04.bst}


\section{Introduction and the Main Result

Taking into account frictions is an important challenge in financial modelling.
In this paper, we focus on the friction that investment decisions may be based only on delayed information,
the actual present market prices being unknown at the time of decision making. In line with other, more widely
investigated frictions like transaction costs, information delay or, more generally, operating under partial information makes the computation
of prices and hedges rather difficult. It is thus of interest to resort to asymptotic methods that study suitably
re-scaled models for vanishing friction.

For the case of transaction costs,
the seminal work of Barles and Soner \cite{BS:98} determines the scaling limit
of utility indifference prices of vanilla options for small proportional transaction costs and high risk aversion.
The present note provides an analogous analysis for the case of information delay,
albeit using convex duality and martingale techniques rather than taking a
PDE approach as pursued in~\cite{BS:98}.

Let us briefly review some of the relevant literature on models with partial information. In \cite{S:94} the author
solved the mean variance hedging problem with partial information for European contingent claims when the asset price process is a martingale. These results were extended beyond the martingale case, for instance, in \cite{F:00,MTT:08} and for defaultable claims in \cite{KX:07}. In a recent paper
\cite{MZS:18} the authors studied the mean variance hedging problem in a setup where the investor has deterministic information
and obtained explicit
solutions in the Bachelier and Black–Scholes models.
The papers \cite{KS:06, CKT:17} develop an arbitrage theory
for hedging with partial information and the papers
\cite{IM:17,DZ:20} treat super-replication with delayed information.
For more details about hedging with partial information see also \cite{Z:17}.
Let us also mention the work
\cite{NO:09} which applied Malliavin calculus
to the utility maximization problem with delayed information. A more recent paper is
\cite{SZ:19} where the authors introduce and study a general framework of optimal stochastic control with delayed information.

To formulate our main result in this note, let
$(\Omega, \mathcal{F}, \mathbb P)$ be  a complete probability space carrying a
one--dimensional Brownian motion
$(W_t)_{t \in [0,T]}$ with natural augmented filtration $(\mathcal{F}^W_t)_{t \in [0,T]}$ and time horizon $T \in (0,\infty)$.
We consider a simple financial market
with a riskless savings account bearing zero interest (for simplicity) and with a risky
asset $S$ with Bachelier price dynamics
\begin{equation*}
S_t=s_0+\sigma W_t+\mu t, \quad t \in [0,T],
\end{equation*}
where $s_0 \in \RR$ is the initial asset price, $\sigma>0$ is the constant volatility and $\mu\in\mathbb{R}$ is the constant drift.

Contrary to the usual setting considered in Mathematical Finance, we focus on an investor who is informed about the risky asset's price changes with a delay $H>0$. As a result, the investor's decisions at time $t$ can
depend only on the information observed up to time $t-H$ as described by the filtration
$\mathcal G^H_t:=\mathcal F^W_{(t-H)^+}$, $t\in [0,T]$.
In line with this delay, the set $\cA^H$ of admissible simple trading strategies consists of
piecewise constant processes of the form
\begin{equation}\label{form}
\gamma_t=\sum_{i=1}^{N} \gamma_{t_i} 1_{(t_{i-1},t_{i}]}(t), \quad  t\in [0,T],
\end{equation}
where $N\in\mathbb N$ is deterministic,
$0=t_0< t_1<\dots< t_N=T$ form a partition of $[0,T]$ and where each
 $\gamma_{t_i}$,  $i=1,\dots,N$, is a $\mathcal G^H_{t_{i-1}}$-measurable random variable.
For any $\gamma\in\mathcal A^H$ of the form~\eqref{form},
the corresponding P\&L at the maturity date $T$ is given as usual:
\begin{equation*}
V^{\gamma}_T:=\sum_{i=1}^{N} \gamma_{t_i}(S_{t_{i}}-S_{t_{i-1}}).
\end{equation*}
The investor will use this investment opportunity to hedge against the payoff of a vanilla European option with payoff $F=f(S_T)$ where
$f:\mathbb R\rightarrow [0,\infty)$ is continuous with strictly sub-quadratic growth in the sense that for some $p\in (0,2)$ there is $C \in [0,\infty)$ with
\begin{equation}\label{2.0+}
0 \leq f(x)\leq C(1+|x|^p),  \quad x\in\mathbb R.
\end{equation}
It is immediate from~\eqref{2.0+} that
\begin{align*}
\mathbb E_{\mathbb P}[\exp\left({\alpha f(S_T)}\right)]<\infty, \quad \alpha\in\mathbb R.
\end{align*}
The investor will assess the quality of a hedge by the resulting expected utility. Assuming exponential utility with constant absolute risk aversion $\lambda>0$, the utility indifference price of
the claim $f(S_T)$ (see, e.g., \cite{R:08} for details on indifference prices) does not depend on the investor's initial wealth
and takes the well-known form
\begin{equation}\label{2.2}
\pi(H,\lambda,f(S_T))=
\frac{1}{\lambda}\log\left(\frac{\inf_{\gamma\in\mathcal A^H}\mathbb E_{\mathbb P}\left[\exp\left(-\lambda\left(V^{\gamma}_T-f(S_T)\right)\right)\right]}
{\inf_{\gamma\in\mathcal A^H}\mathbb E_{\mathbb P}\left[\exp\left(-\lambda V^{\gamma}_T\right)\right]}\right).
\end{equation}
\begin{rem}\label{rem:denom}
 The infimum in the denominator of~\eqref{2.2} is well-known to be attained by the constant strategy $\gamma \equiv \mu/(\lambda \sigma^2)$ which of course is admissible with any delay in information and which, for any risk aversion $\lambda>0$, yields the same value $\exp(-(\mu^2T/(2\sigma^2))$. As a consequence, the relevant quantity to be understood is the numerator in~\eqref{2.2}.
\end{rem}
It is readily seen that for $H \downarrow 0$, the above indifference price converges to the usual indifference price with no delay in information. A more interesting limit emerges, however, if we re-scale the investor's risk-aversion in the form $A/H$ as described by our main result which will be proved in Section~\ref{sec:3}:
\begin{thm}\label{thm.1}
For vanishing delay $H \downarrow 0$ and re-scaled high risk-aversion $A/H$ with $A>0$ fixed, the utility indifference price of $f(S_T)$ with continuous $f \geq 0$ exhibiting sub-quadratic growth~\eqref{2.0+}  has the scaling limit
\begin{equation}\label{2.3}
 \lim_{H\downarrow 0} \pi(H,A/H,f(S_T))=
 \sup_{\nu\in\Gamma}\mathbb E_{\mathbb P}\left[ f\left(s_0+\int_{0}^T \nu_t dW_t\right)- \frac{1}{2A\sigma^2}\int_{0}^T\left(\nu_t-\sigma\right)^2dt \right],
\end{equation}
where $\Gamma$ is the class of all bounded  $\mathcal(\cF^W_t)$-predictable processes $\nu$.
\end{thm}
\begin{rem}
The linear re-scaling of risk aversion $A/H$ is in line with the scaling-limit result for super-replication with delay \cite{DZ:20} where it was shown that the delay has to be re-scaled linearly in the time step.
A more formal reason for the scaling
$\frac{A}{H}$ can be found via duality with entropy minimization and the bound given by Lemma~\ref{lem:entropybound}.
\end{rem}
Next, as a corollary of Theorem \ref{thm.1} we obtain the following super-replication result which is the Bachelier version of Theorem 2.1 in \cite{DZ:20}.
\begin{cor}\label{cor.referee}
For any delay $H>0$, the super--replication price of $f(S_T)$
is given by
$\hat f(s_0)$
where $\hat f:\RR \to [0,\infty]$ denotes
the concave envelope of $f$. Moreover,
if $\hat f(s_0)<\infty$ then the buy-and-hold strategy $\gamma\equiv \partial_{+}\hat{f}(s_0)$ is an optimal super-hedge.
\end{cor}
\begin{proof}
First, if $\hat f(s_0)<\infty$ then from concavity $\hat f(s)<\infty$ everywhere and we obtain
$$\hat f(s_0)+\partial_{+}\hat{f}(s_0)(S_T-s_0)\geq\hat f(S_T)\geq f(S_T) \ \ \mbox{a.s.}$$
In other words the strategy with initial capital $\hat f(s)$ and
$\gamma\equiv \partial_{+}\hat{F}(s)$
is a super-hedge. To complete the proof it thus remains to establish that
the super--replication price for delay $H>0$, which we denote by $\Pi(H)$, is no less than $\hat f(s_0)$. Clearly, $\Pi(H)$ is non-increasing in $H$ and any super-replication price is larger than the corresponding utility indifference price. In particular, we have $\Pi(H)\geq \Pi(h)\geq \pi(h,A/h,F)$ for any $h<H$ and $A>0$.
Theorem \ref{thm.1} allows us to let $h \downarrow 0$ and then $A \uparrow \infty$ in this inequality to conclude
\begin{align*}
 \Pi(H) &\geq \lim_{A \uparrow \infty}  \sup_{\nu\in\Gamma}\mathbb E_{\mathbb P}\left[ f\left(s_0+\int_{0}^T \nu_t dW_t\right)- \frac{1}{2A\sigma^2}\int_{0}^T\left(\nu_t-\sigma\right)^2dt \right]\\
 &=\sup_{\nu\in\Gamma}\mathbb E_{\mathbb P}\left[ f\left(s_0+\int_{0}^T \nu_t dW_t\right)\right]=\hat f(s_0),
\end{align*}
where the last equality is well-known in stochastic control; see, e.g., \cite{N:18}, Lemma~3.3 for a proof.
\end{proof}

\begin{rem}
Our otherwise standing continuity and growth assumptions on $f$
can be relaxed in Corollary~\ref{cor.referee} to just lower-semicontinuity. Indeed, the proposed buy-and-hold strategy is then still a super-hedge. To obtain its optimality, we consider an increasing sequence of continuous and bounded payoff profiles $f_n:\mathbb R\rightarrow [0,\infty)$ converging to $f$ from below. Their super-replication prices are obviously not larger than that of $f$ and, by the already established version of Corollary~\ref{cor.referee}, they are given by their concave envelopes
$\hat{f}_n(s_0)\uparrow\hat{f}(s_0)$.
\end{rem}

The proof of Theorem~\ref{thm.1} rests on the duality theory of exponential utility which we will develop for discrete-time information delayed models in Section~\ref{sec:duality}. As in~\cite{KS:06}, it identifies dual probability measures via a relaxation of the martingale property. A lower bound for the scaling limit is then obtained in Section~\ref{sec:lowerbound} by constructing suitable relaxed martingale measures which asymptotically generate a prescribed volatility profile $\nu$. Of course, the entropy of these measures relative to $\P$ explodes (unless $\nu = \sigma$), but we show in~\eqref{eq:333} that the re-scaled risk aversion compensates this adequately in our scaling limit. An upper bound is established in Section~\ref{sec:upperbound} by a suitable lower bound for the entropy of such a relaxed measure which is argued similar to It\^{o}'s isometry, albeit only aggregating 2nd moments from sufficiently distant increments in a carefully chosen way.

Let us finish this section with the following two remarks on how one could compute the right-hand side of~\eqref{2.3}.
\begin{rem}\begin{enumerate}
\item \emph{Connection with optimal transport.} If the distribution of the random variable
$s+\int_{0}^T \nu_u dW_u$
is fixed then from the It\^{o} Isometry we get that the optimal $\nu$
in the right hand side of~\eqref{2.3} is the one which maximizes (under the constraint on the distribution
of $s+\int_{0}^T \nu_u dW_u$)
the expectation $\mathbb E_{\mathbb P}\left[\int_{0}^T \nu_u du\right]$. From Theorem~1.10
in \cite{BBHK:20} we get that the optimal $\nu$ integrates to a random variable of the form
$\int_{0}^T \nu_u dW_u=\zeta(W_T)$
for some appropriate $\zeta:\mathbb R\rightarrow\mathbb R$.
Thus, letting $\mu\sim W_T$ denote, the normal distribution with mean zero and variance $T$, It\^{o}'s representation theorem isometry show that the right hand side of~\eqref{2.3} can be represented as
$$\sup_{\zeta:\mathbb R\rightarrow\mathbb R \text{ with } \int_{\mathbb R} \zeta(z)\mu(dz)=0 } \int_{\mathbb R} \left(f(s+\zeta(z))-\frac{1}{2A\sigma^2}[\zeta(z)-\sigma z]^2\right)\mu(dz).$$
This is a deterministic optimization problem which can be analyzed with
the classical techniques of calculus of variations.

\item \emph{Connection with nonlinear PDE.} The right-hand side of~\eqref{2.3} can also be viewed as the Bachelier version of the stochastic control problem
which is given by the right-hand side of formula~(3) in
\cite{L:18}. That paper studies
hedging of a European option in the Black--Scholes model with linear price impact and describes the option's replication price  by a stochastic control problem similar to the right hand side of~\eqref{2.3}.
Moreover, the author studies in detail the PDE which corresponds to such a control problem. We have to leave open how to give a financial interpretation of the similarity between the mathematical form of our scaling limit representation~\eqref{2.3} and the
replication price formula of \cite{L:18}.
\end{enumerate}
\end{rem}

\section{Proof of Theorem \ref{thm.1}}\label{sec:3}

The proof of Theorem~\ref{thm.1} will be done by estimating the scaling $\liminf$ from below in Section~\ref{sec:lowerbound} and the scaling $\limsup$ from above in~Section~\ref{sec:upperbound}. Both sections rely on duality results for discrete-time models with information delay that we collect in the preparatory Section~\ref{sec:duality} that we turn to next.

\subsection{Duality in models with information delay}\label{sec:duality}

The proof of Theorem~\ref{thm.1} relies on the duality theory for exponential utility maximization with information delay. We will combine results from \cite{KS:06} and \cite{Fritelli:00} to this end.

Taking an abstract view, let us consider a price process $S$ adapted to a filtration $(\cF_t)$ and an investor who will make investment decision using a filtration $(\cG_t)$ which may be strictly contained in $(\cF_t)$ at any time. The payoff will also be given by a general random variable $F$.

\begin{lem}\label{lem:easydualitybound}
 Let $\gamma$ be a $(\cG_t)_{t \in \TT}$-predictable strategy that changes value only at times in $\TT=\{0=t_0<\dots<t_N\leq T\}$ and suppose that $\E_{\P}[\exp(\lambda |F|)]<\infty$. Then
 \begin{align}\label{eq:53}
     \frac{1}{\lambda} \log \E_{\P}\left[\exp\left(-\lambda(V^\gamma_T-F)\right)\right] \geq \E_{\QQ}\left[F-\frac{1}{\lambda} \log \frac{d\QQ}{d\P}\right]
 \end{align}
 for any probability $\QQ\sim\P$ with finite entropy $\E_{\QQ}[\log \frac{d\QQ}{d\PP}]<\infty$ relative to $\P$ such that for all $u \in \TT$  we have $S_u \in L^1(\QQ)$ and
 \begin{align}\label{eq:51}
     \E_{\QQ}\left[S_t-S_s\middle| \cG_s\right]=0 \text{ for any } s,t \in \TT \text{ with } s \leq t.
 \end{align}
\end{lem}
\begin{rem}
 \cite{KS:06} show that existence of a probability measure $\QQ\sim\P$ with the relaxed martingale property~\eqref{eq:51} is equivalent to the absence of arbitrage when investors are confined to $(\cG_t)$-predictable strategies acting only at times from $\TT$. The relaxation accounts for the information differential between the filtration $(\cF_t)$ to which $S$ is adapted and the filtration $(\cG_t)$ that the investor's trades are based upon.
\end{rem}
As usual, the proof rests on the classical Legendre-Fenchel duality inequality
\begin{align}\label{eq:2}
    xy \leq e^x + y(\log y-1), \quad x \in \RR, y>0, \quad \text{ with `$=$' iff } y=e^x.
\end{align}
For instance, using this inequality with $x=\lambda F$, $y=d\QQ/d\P$ yields
\begin{align*}
    \lambda F \frac{d\QQ}{d\P} \leq e^{\lambda F} + \frac{d\QQ}{d\P}\log \frac{d\QQ}{d\P} \in L^1(\P)
\end{align*}
so that the right-hand side in~\eqref{eq:53} is always well-defined under our assumptions on $F$ and $\QQ$.
\begin{proof}

Without loss of generality we can assume that the expectation on the left-hand side of~\eqref{eq:53} is finite. Note that then by~\eqref{eq:2} and the Cauchy-Schwarz inequality
\begin{align*}
    \E_\QQ\left[\frac{\lambda}{2} (V^\gamma_T)^-\right]
    &=\E_\P\left[\frac{\lambda}{2} (V^\gamma_T)^-\frac{d\QQ}{d\P}\right]
    \leq \E_\P\left[e^{\frac{\lambda}{2} (V^\gamma_T)^-}+\frac{d\QQ}{d\P}\left(\log\frac{d\QQ}{d\P}-1\right)\right]\\    &\leq 1+\E_\P\left[e^{-\frac{\lambda}{2} (V^\gamma_T-F)}e^{-\frac{\lambda}{2}F}\right]+\E_{\QQ}\left[\log\frac{d\QQ}{d\P}-1\right]\\
    &\leq \left(\E_\P\left[e^{-\lambda (V^\gamma_T-F)}\right]\right)^{1/2}\left(\E_{\P}\left[e^{-{\lambda}F}\right]\right)^{1/2}+\E_{\QQ}\left[\log\frac{d\QQ}{d\P}\right]
\end{align*}
is finite under our assumptions on $F$ and $\QQ$.

We will next use a variant of a classical martingale transform argument (see, e.g., \cite{FollmerSchied:16}, Theorem~5.14) to conclude from this that actually $V^\gamma_T$ is $\QQ$-integrable with $\E_\QQ[V^\gamma_T]=0$. Indeed, using (a priori) generalized conditional expectations (as defined in~\cite{S}), we can now deduce by Jensen's inequality that $\hat{V}_{T}:=\E_\QQ\left[V^\gamma_T\middle|\cG_T\right]$ exists with $\hat{V}_{T}^- \in L^1(\QQ)$. Hence, $\hat{V}_{t_{N-1}}:=\E_\QQ\left[\hat{V}_{t_N}\middle|\cG_{t_{N-1}}\right]$ exists which, due to~\eqref{eq:51}, can be seen to be equal to $\E_\QQ\left[V^\gamma_{t_{N-1}}\middle|\cG_{t_{N-1}}\right]$; by Jensen's inequality we have again $\hat{V}_{t_{N-1}}^-\in L^1(\QQ)$. We can now proceed inductively to conclude that $\hat{V}_0:=\E_\QQ\left[\hat{V}_{t_1}\middle|\cG_0\right]$ exists and coincides with
$\E_{\QQ}[V^\gamma_0]=0$. Reversing the course of our induction argument, it follows that $\hat{V}_{t_1}$ is $\QQ$-integrable with vanishing expectation, a property that inductively follows finally also for $\hat{V}_{T}$ and, thus, for $V^\gamma_T$ itself.

To conclude our claim~\eqref{eq:53} we can now use~\eqref{eq:2} again to find for any $y>0$ the estimate
\begin{align*}
    \E_{\P}\left[\exp\left(-\lambda(V^\gamma_T-F)\right)\right]
    &=\E_{\P}\left[\left(e^{-\lambda V^\gamma_T}+\lambda V^\gamma_T y e^{-\lambda F} \frac{d\QQ}{d\P}\right)e^{\lambda F}\right] \\
    &\geq \E_{\P}\left[\left(-y e^{-\lambda F}\frac{d\QQ}{d\P}\left(\log \left(ye^{-\lambda F} \frac{d\QQ}{d\P}\right)-1\right)\right)e^{\lambda F}\right] \\
    &= -y\left(\log y -1\right)+\E_{\QQ}\left[\lambda F-\log \frac{d\QQ}{d\P}\right]y.
\end{align*}
Using that due to~\eqref{eq:2} the supremum over $y>0$ in  the last expression is $e^x$ for $x=\E_{\QQ}\left[\lambda F-\log \frac{d\QQ}{d\P}\right]$  yields the result.
\end{proof}

The above duality estimate turns out to be tight in the sense that there is no duality gap  at least for discrete-time settings:
\begin{thm}\label{thm:duality}
 For $\TT=\{0=t_0<\dots<t_N\leq T\}$, let $\cA(\TT)$ be the set of $(\cG_t)_{t \in \TT}$-predictable strategies that only trade at times from $\TT$ and let $\cQ(\TT)$ denote the set of measures $\QQ\sim \P$ with bounded entropy relative to $\P$ that exhibit the relaxed martingale property~\eqref{eq:51} for $S$. Suppose in addition that there is $\epsilon>0$ such that $\E_{\P}[\exp(\lambda(1+\epsilon) |F|)]<\infty$ and $\E_{\P}[\exp(\epsilon|S_t|^{1+\epsilon})]<\infty$, $t \in \TT$. Finally, assume absence of arbitrage in the relaxed sense that $\cQ(\TT)\not=\emptyset$.

 Then the certainty equivalent of $F$ has the dual representation
  \begin{align}\label{eq:61}
    \inf_{\gamma \in \cA(\TT)} \frac{1}{\lambda} \log \E_{\P}\left[\exp\left(-\lambda(V^\gamma_T-F)\right)\right] = \sup_{\QQ \in \cQ(\TT)} \E_{\QQ}\left[F-\frac{1}{\lambda} \log \frac{d\QQ}{d\P}\right]
 \end{align}
 and the latter supremum is actually attained.
\end{thm}
\begin{rem}
The literature on exponential utility maximization mostly considers (locally) bounded asset prices and claims and provides examples for non-existence of an entropy-minimizing martingale measure if these conditions are violated; see, e.g., \cite{Fritelli:00}. In particular it does not cover the discrete-time version of our Bachelier model. Neither does this literature cover information delay. So, even though the above result is not particularly surprising, we include it for the sake of completeness.
\end{rem}
\begin{proof}
That `$\geq$' holds true in~\eqref{eq:61} is immediate from Lemma~\ref{lem:easydualitybound}.
Let us next observe that it is sufficient to treat the case $\lambda =1$ by passing to $\lambda F$ and that without loss of generality we can assume $F=0$ by passing from $\P$ to $\P^F$ with $d\P^F/d\P = e^{\lambda F}/\E_{\P}[e^{\lambda  F}]$ for which  $$\E_{\QQ}\left[\log \frac{d\QQ}{d\P^F}\right]=\E_{\QQ}\left[\log \frac{d\QQ}{d\P}-\lambda F\right],  \quad \QQ \in \cQ(\TT);$$
moreover, we have $\E_{\P^F}[\exp(\epsilon'|S_t|^{1+\epsilon'})] <\infty$ for any $\epsilon' \leq \epsilon/(1+\epsilon)$ by virtue of H{\"o}lder's inequality and the exponential integrability assumptions on $S$ and $F$ under $\P$.

To prove that the supremum in~\eqref{eq:61} is attained, we use the well-known Komlos-argument to obtain a maximizing sequence $\QQ_n \in \cQ(\TT)$, $n=1,2,...$, for which $Z_n := d\QQ_n/d\P$ converges almost surely; in fact, convergence holds also in $L^1(\P)$ because $H(\QQ_n|\P):=\E_\P\left[Z_n\log Z_n\right]=\E_{\QQ_n}\left[\log \frac{d\QQ_n}{d\PP}\right]$ can be assumed bounded in $n=1,2,\dots$ without loss of generality. Hence, $Z_0 := \lim_n Z_n$ yields the density of a probability measure $\QQ_0$ which by Fatou's lemma satisfies $H(\QQ_0|\P)\leq \lim_n H(\QQ_n|\P)$. So, attainment of the supremum in~\eqref{eq:61} is proven if we can argue that $\QQ_0$ is a martingale measure for $S$ in the relaxed sense of~\eqref{eq:51}. For this we show below that
\begin{align}\label{eq:4}
    R_M:= \max_{t \in \TT} \sup_{n=0,1,\dots}  \E_{\QQ_n}[|S_t|1_{\{|S_t|>M\}}] \to 0 \text{ as } M\uparrow \infty.
\end{align}
We then can estimate, for any $A \in \cF$ and $t \in \TT$,
\begin{align*}
    \left|\E_{\QQ_n}\left[S_t1_A\right]\right.&\left.-\E_{\QQ_0}\left[S_t1_A\right]\right| \\& \leq  \left|\E_{\QQ_n}\left[(-M \vee S_t \wedge M)1_A\right]-\E_{\QQ_0}\left[(-M \vee S_t \wedge M)1_A\right]\right|+ 2R_M
\end{align*}
to conclude from $Z_n \rightarrow Z_0$ in $L^1(\P)$ that
\begin{align*}
    \E_{\QQ_n}\left[S_t1_A\right] \xrightarrow[n \uparrow \infty]{} \E_{\QQ_0}\left[S_t 1_A\right], \quad A \in \cF.
\end{align*}
Exploiting the relaxed martingale property of $S$ under each of the $\QQ_n$, we conclude for any $t \in \TT$ and $A \in \cG_t$ the required martingale identity
\begin{align*}
    \E_{\QQ_0}[S_T1_A]=\lim_n \E_{\QQ_n}[S_T1_A]= \lim_n \E_{\QQ_n}[S_t1_A] =  \E_{\QQ_0}[S_t1_A].
\end{align*}

It remains to prove~\eqref{eq:4}. For this we let $q > 1$ be the conjugate of $p>1$ with $1/p+1/q=1$ to apply H{\"o}lder's inequality:
\begin{align}
    \E_{\QQ_n}[|S_t|1_{\{|S_t|>M\}}] &= \E_{\P}[(Z_n)^{1/p}|S_t| (Z_n)^{1/q}1_{\{|S_t|>M\}}] \nonumber \\&\label{eq:5}
    \leq (\E_{\P}[Z_n |S_T|^p])^{1/p}(\E_{\QQ_n}[1_{\{|S_t|>M\}}])^{1/q}.
\end{align}
Now, we recall $\epsilon>0$ from  our exponential integrability assumptions on $|S|^{1+\epsilon}$ and choose $p:=1+\epsilon$. Due to the Legendre-Fenchel duality estimate~\eqref{eq:2}, we have
\begin{align*}
    Z_n \epsilon |S_t|^p \leq e^{\epsilon |S_t|^{1+\epsilon}} + Z_n (\log Z_n-1)
\end{align*}
and we can thus bound the first expectation in~\eqref{eq:5} uniformly in $n=0,1,\dots$. Due to $\E_{\QQ_n}[1_{\{|S_t|>M\}}] \leq \E_{\QQ_n}[|S_t|]/M=\E_{\P}[Z_n |S_t|]/M$, the second one vanishes uniformly in $n=0,1,\dots$ for $M \uparrow \infty$ because we already have shown that $(Z_n|S_t|)_{n=1,2,\dots}$  is bounded in $L^1(\P)$, which by Fatou's lemma also yields $\E_{\QQ_0}[|S_t|]=\E_{\P}[Z_0|S_t|]<\infty$.  Since $\TT$ is finite, it follows that $\lim_M R_M = 0$ as claimed.

The proof of `$\leq$' in~\eqref{eq:61} can now be done following the approach in \cite{Fritelli:00} that also works with the relaxed martingale condition rather than the usual one. Indeed, as outlined in the proof of Theorem~2.3 there, the first order condition for the entropy minimizing measure reveals its density to be of the form $Z_0=\exp(-f_0)/\E_{\P}[\exp(-f_0)]$ for some random variable $f_0$ with $\E_{\QQ_0}[f_0]=0$ and $\E_\QQ[f_0]\leq 0$ for any $\QQ \in \cQ(\TT)$. The separation argument for Theorem~2.4 of \cite{Fritelli:00} then shows that $f_0$ is contained in the $L^1(\QQ)$-closure of $\{V^\gamma_T-L^\infty_+\;|\; \gamma \text{ bounded, $(\cG_t)_{t \in \TT}$ predictable}\}$ for any $\QQ \in \cQ(\TT)$. In particular there are $\gamma_n$ as considered in this set and $R_n \in L^\infty_+$, $n=1,2,\dots$, such that $f_0=\lim_n (V^{\gamma_n}_T-R_n)$ in probability. Since $\cQ(\TT)\not=\emptyset$, Theorem~1 in \cite{KS:06} yields that $f_0$ must be of the same form $f_0=V^{\gamma_0}_T-R_0$ for some $(\cG_t)_{t \in \TT}$-predictable $\gamma_0$ and some $R_0 \geq 0$. As a result, the left-hand side in~\eqref{eq:61} is less than or equal to
\begin{align*}
    \log \E_{\P}[\exp(- V^{\gamma_0}_T)] \leq  \log \E_{\P}[\exp(-f_0)].
\end{align*}
By construction of $\QQ_0$ and $f_0$ the right-hand side is equal to
\begin{align*}
    - \E_{\QQ_0} [\log Z_0] =\E_{\QQ_0}\left[f_0+\log \E_{\P}[\exp({-f_0})]\right]=\log \E_{\P}[\exp({-f_0})],
\end{align*}
which coincides with the upper bound we just have found for the left hand side. This yields the desired `$\leq$'-relation in~\eqref{eq:61}.
\end{proof}

\subsection{Lower bound for the scaling limit}\label{sec:lowerbound}

To prove `$\geq$' in~\eqref{2.3} we construct for vanishing time delay $H_n \downarrow 0$ relaxed martingale measures $\QQ_n$, $n=1,2,\dots$, under which $S_T$ converges in law to $s+\int_0^T \nu_u dW_u$ while their relative entropy with respect to $\P$ explodes at most like $\E_{\P}\left[\int_0^T(\nu_u-\sigma)^2du/(2\sigma^2H_n)\right]$.

To this end observe first that, by standard density arguments, we will get the same supremum there if instead of letting $\nu$ vary over all of $\Gamma$ there, we confine it to be   of the form
$\nu = \hat{\nu}(W)$ with
\begin{align*}
\hat{\nu}_t(x) := \sum_{j=1}^{J-1} f_j(x_{t_0},\dots,x_{t_{j-1}}) 1_{[t_{j},t_{j+1})}(t),  \quad t\in [0,T], \; x \in C[0,T],
  \end{align*}
where $0=t_0<t_1<\dots<t_J=T$ is a finite deterministic partition of $[0,T]$
and each $f_j:\mathbb R^{j}\rightarrow\mathbb R$, $j=1,...,J$, is Lipschitz continuous and bounded. Thus, it suffices to show that each such $\nu$ gives a lower bound for the scaling limit in~\eqref{2.3} along a fixed sequence $H_n \downarrow 0$:
\begin{equation}\label{3.6}
 \liminf_{n\rightarrow\infty} \pi(H_n,A/H_n,f(S_T))\geq
\mathbb E_{\mathbb P}\left[ f\left(s+\int_{0}^T \nu_t dW_t\right)- \frac{1}{2A\sigma^2}\int_{0}^T\left(\nu_t-\sigma\right)^2dt \right].
\end{equation}
For $n=1,2,\dots$, consider the SDE
\begin{align*}
    X^{n}_0=0,\quad
dX^{n}_t=dW_t+\kappa^n_t(X^n)dt
\end{align*}
with the path-dependent drift-coefficient
\begin{align*}
\kappa^n_t(x) &:= \frac{1}{\sigma}\left(\mu-\sum_{j=1}^{J-1}1_{[t_{j},t_{j+1})}(t)\frac{f_j(x_{t_0},\dots,x_{t_{j-1}})-\sigma}{H_n}\Phi\left(x_{t}-x_{(t-H_n)\vee t_{j}}\right)\right)
\end{align*}
where $\Phi(z):=(-1)\vee z \wedge 1$. Since $\kappa^n$ is Lipschitz continuous in the supremum norm on $C[0,T]$, Theorem~2.1 from Chapter IX in \cite{RY} yields a unique strong solution for this SDE. Moreover, boundedness of $\kappa^n$ ensures via Girsanov's theorem that
\begin{align}\label{eq:22}
    \frac{d\QQ_n}{d\P}:= \cE\left(-\int_0^. \kappa^n_u(X^n)dW_u\right)_T
\end{align}
yields the density of a measure $\QQ_n \sim \P$ under which $X^n$ is a Browninan motion. Hence,
\begin{align}\label{eq:21}
    S_t = s+\sigma X^n_t + \int_0^t  (\mu-\sigma \kappa^n_u(X^n)) du
\end{align}
is $\QQ_n$-integrable. Without loss of generality we can assume that $t_{j-1}\leq (t_j-H_n)^+$ for any $j=1,\dots,J$ and so, for any $t\in[t_{j},t_{j+1}]$, we get furthermore
\begin{align*}
    \E_{\QQ_n}&\left[S_{t_{j+1}}-S_t\middle| \cG^{H_n}_t \right] \\=& \int^{t_{j+1}}_t\frac{f_j(X^n_{t_0},\dots,X^n_{t_{j-1}})-\sigma}{H_n}\E_{\QQ_n}\left[ \Phi\left(X^n_{u}-X^n_{(u-H_n)\vee t_{j}}\right) \middle| \cF^W_{(t-H_n)^+}  \right]du.
\end{align*}
By symmetry of both $\Phi$ and the increments of Brownian motion, the latter conditional expectations all vanish and it thus follows that each $\mathbb Q_n$ exhibits the relaxed martingale property~\eqref{eq:51} for $\TT=[0,T]$ and $\cG_t=\cG^{H_n}_t$, $t \in [0,T]$.

Next, from \eqref{eq:21} and letting $\Psi(z):=\Phi(z)-z$, $z \in \RR$, we obtain that, for any $j=0,\dots,J-1$,
\begin{align*}
S_{t_{j+1}}-S_{t_j}=\;&
\sigma (X^{n}_{t_{j+1}}-X^{n}_{t_{j}})+\frac{\hat\nu_{t_j}(X^n)-\sigma}{H_n}\int_{t_j}^{t_{j+1}} \left(X^{n}_u-X^{n}_{(u-H_n)\vee t_j}\right) du\\
&+\frac{\hat\nu_{t_j}(X^n)-\sigma}{H_n}\int_{t_j}^{t_{j+1}} \Psi\left(X^{n}_u-X^{n}_{(u-H_n)\vee t_j}\right) du\\
=\;&\hat\nu_{t_j}(X^n) (X^{n}_{t_{j+1}}-X^{n}_{t_{j}})\\
&+\frac{\hat\nu_{t_j}(X^n)-\sigma}{H_n}\int_{t_{j+1}-H_n}^{t_{j+1}} (X^{n}_u-X^{n}_{t_{j+1}}) du\\
&+\frac{\hat\nu_{t_j}(X^n)-\sigma}{H_n}\int_{t_j}^{t_{j+1}} \Psi\left(X^{n}_u-X^{n}_{(u-H_n)^{+}}\right) du.
\end{align*}
Now take the sum over $j$. Since $X^n$ is a $\QQ_n$-Brownian motion and $|\Psi(z)| \leq z^4$, the contributions from the last two summands converge to 0 in distribution and the contribution from the first one converges in distribution to $\int_0^T \hat{\nu}_t(W)dW_t$. Hence, we can use Slutsky's theorem to conclude that
\begin{align*}
    \Law(S_T|\QQ_n) \to \Law\left(s+\int_0^T \nu_t dW_t\;\middle|\;\P\right)
\end{align*}
and, so, Fatou's lemma yields the lower bound
\begin{equation}\label{3.9}
\liminf_{n\rightarrow\infty}\mathbb E_{\mathbb Q_n}[f(S_T)]\geq\mathbb E_{\mathbb P}\left[f\left(s+\int_{0}^T\nu_t dW_t\right)\right].
\end{equation}

Finally, recalling the construction~\eqref{eq:22} of $\QQ_n$ we get
\begin{align*}
    \lim_n & \left(H_n\mathbb E_{\mathbb Q_n}\left[\log\left(\frac{d\mathbb Q_n}{d\mathbb P}\right)\right]\right)
=\lim_n \left(\frac{H_n}{2}\mathbb E_{\mathbb Q_n}\left[\int_{0}^T |\kappa^{(n)}_t|^2 dt\right]\right)\\
&=\lim_n \left(\sum_{j=1}^{J-1}
\mathbb E_{\mathbb Q_n}\left[\frac{(\hat\nu_{t_j}(X^n)-\sigma)^2 }{2\sigma^2 H_n}\int_{t_j}^{t_{j+1}}\mathbb E_{\mathbb Q_n}\left[\Phi^2\left(X^{n}_t-X^{n}_{(t-H_n)\vee t_j}\right)\middle|\mathcal F^W_{t_j}\right]dt\right]\right)\\
&=\lim_n \left(\sum_{j=1}^{J-1}
\mathbb E_{\mathbb P}\left[\frac{(\hat\nu_{t_j}(W)-\sigma)^2 }{2\sigma^2}\int_{t_j}^{t_{j+1}}\mathbb E_{\mathbb P}\left[(W_1)^2 \wedge \frac{1}{H_n}\right]dt\right]\right)\\
&=\sum_{j=1}^{J-1}\mathbb E_{\mathbb P}\left[\frac{(\nu_{t_j}-\sigma)^2}{2\sigma^2}(t_{j+1}-t_j)\right]
\end{align*}
where the last but one equality holds because $X^n$ is a $\QQ_n$-Brownian motion.
We conclude that
\begin{align}\label{eq:333}
\lim_{n\rightarrow\infty}\left(H_n\mathbb E_{\mathbb Q_n}\left[\log\left(\frac{d\mathbb Q_n}{d\mathbb P}\right)\right]\right)=
\frac{1}{2\sigma^2}\mathbb E_{\mathbb P}\left[\int_{0}^T\left(\nu_t-\sigma\right)^2dt \right].
\end{align}
This together with Lemma~\ref{lem:easydualitybound}
and \eqref{3.9} gives the desired lower bound \eqref{3.6} that we wanted to establish in this section.

\subsection{Upper bound for the scaling limit}\label{sec:upperbound}

For the upper bound of the scaling limit, we will need a lower bound for the relative entropy of a relaxed martingale measure with respect to $\P$ that we establish first:

\begin{lem}\label{lem:entropybound}
 For any delay $H>0$ and $M=1,2,\dots$, the relative entropy of a measure $\QQ \sim \P$ satisfying the relaxed martingale condition~\eqref{eq:51} over $\TT_{H/M}:=\{kH/M:k=0,1,\dots\} \cap [0,T]$ is bounded from below by
 \begin{align*}
     \E_{\QQ}\left[\log \frac{d\QQ}{d\PP}\right] \geq \frac{1}{2\sigma^2H} \frac{M}{M+1}  \E_{\QQ}\left[\left(S_{T_{H/M}}-s-\sigma W^\QQ_{T_{H/M}}\right)^2\right]
 \end{align*}
 where $T_{H/M}:=\max \TT_{H/M}$ and where $W^\QQ$ is the $\QQ$-Brownian motion induced by $W$ via Girsanov's theorem.
\end{lem}
\begin{proof}
 By It\^{o}'s representation theorem $W^\QQ = W-\int_0^. \alpha_u du$ for the predictable $\alpha$ such that
 \begin{align*}
     \left.\frac{d\QQ}{d\PP}\right|_{\cF^W_t}=\cE\left(\int_0^. \alpha_u dW_u\right)_t, \quad t \in [0,T].
 \end{align*}
 Hence, the relative entropy to be bounded from below is
 \begin{align*}
     \E_{\QQ}\left[\log \frac{d\QQ}{d\PP}\right] =\frac{1}{2}\E_{\QQ}\left[\int_0^T \alpha^2_u du\right]
 \end{align*}
 and
 \begin{align*}
     S_t-s-\sigma W^\QQ_t = \int_0^t \sigma \alpha_u du + \mu t, \quad t \in [0,T].
 \end{align*}

Fix $M\in\{1,2,\dots\}$ and $H>0$ and let us consider the increments over periods of length $H/M$:
\begin{align*}
    \Delta_t&:=S_{(t+H/M) \wedge
    T}-S_{t\wedge
    T}-\sigma\left(W^\QQ_{(t+H/M)\wedge
    T}-W^\QQ_{t\wedge
    T}\right)\\&= \int_{t\wedge
    T}^{(t+H/M)\wedge
    T}  (\sigma\alpha_u+\mu) du.
\end{align*}
Observe that
\begin{align}\label{eq:41}
S_{T_{H/M}}-s-\sigma W^{\QQ}_{T_{H/M}}=\sum_{k=1}^{K_{H/M}}    \Delta_{(k-1)H/M}
\end{align}
for $K_{H/M}:=T_{H/M}/(H/M)$. Moreover, due to the relaxed martingale property of $S$ (and the standard martingale property of $W^{\QQ}$) under $\QQ$, we have
\begin{align}\label{eq:42}
    \E_{\QQ}[\Delta_t]=0 \quad \text{ and } \quad  \E_{\QQ}[\Delta_t\Delta_s]=0
\end{align}
for any $t \in [0,T]$ and any $s \in [0,T]$ with $|t-s|\geq (M+1)H/M$.

Hence, subdividing $[0,T]$ into intervals of length $H/M$ we can estimate by Jensen's inequality
\begin{align*}
      \E_{\QQ}\left[\log \frac{d\QQ}{d\PP}\right] &= \frac{1}{2}\sum_{k=1}^\infty  \E_{\QQ}\left[\int_{((k-1)H/M)\wedge T}^{(kH/M)\wedge T} \alpha^2_u du\right] \\&\geq \frac{1}{2}\sum_{k=1}^{K_{H/M}} \E_{\QQ}\left[\frac{1}{H/M}\left(\int_{((k-1)H/M}^{kH/M} \alpha_u du\right)^2\right]\\
      &= \frac{M}{2H}\sum_{k=1}^{K_{H/M}} \E_{\QQ}\left[\left(\frac{\Delta_{(k-1)H/M}-\mu H/M}{\sigma}\right)^2\right]
      \\&=  \frac{M}{2H\sigma^2}\sum_{k=1}^{K_{H/M}}\left( \E_{\QQ}\left[\left(\Delta_{(k-1)H/M}\right)^2\right]+(\mu H/M)^2\right),
  \end{align*}
where in the last step we used $\E_{\QQ}[\Delta^M_t]=0$. Dropping the squares involving $\mu$ and rearranging the sum over $k$ gives
\begin{align*}
   \E_{\QQ}\left[\log \frac{d\QQ}{d\PP}\right] & \geq  \frac{M}{2H\sigma^2}\sum_{i=0}^M \sum_{j:i+j(M+1)< K_{H/M}} \E_{\QQ}\left[\left(\Delta_{(i+j(M+1)H/M}\right)^2\right].
\end{align*}
In this rearrangement, we have due to~\eqref{eq:42} that for each $i=0,\dots,M-1$ the sums over $j$  are taken over $\QQ$-uncorrelated increments and therefore amount to
\begin{align*}
    \E_{\QQ}\left[\sum_{j} \left(\Delta_{(i+j(M+1))H/M)}\right)^2\right]
    =\E_{\QQ}\left[\left(\sum_{j} \Delta_{(i+j(M+1))H/M)}\right)^2\right].
\end{align*}
Plugging this into our above estimate yields
\begin{align*}
    \E_{\QQ}\left[\log \frac{d\QQ}{d\PP}\right] & \geq  \frac{M}{2H\sigma^2}  \E_{\QQ}\left[\sum_{i=0}^M\left(\sum_{j:i+j(M+1)< K_{H/M}}\Delta_{(i+j(M+1)H/M}\right)^2\right]\\
    & \geq \frac{M}{2H\sigma^2}   \E_{\QQ}\left[\frac{1}{M+1}\left(\sum_{i=0}^M\sum_{j:i+j(M+1)< K_{H/M}}\Delta_{(i+j(M+1)H/M}\right)^2\right]
\end{align*}
where the last estimate is due to Jensen's inequality. The above double sum is aggregating all increments $\Delta$ over intervals $[(k-1)H/M,kH/M]$ fully fitting into $[0,T]$ and thus coincides with the left-hand side of~\eqref{eq:41}. We obtain the claimed lower bound for the entropy.
\end{proof}

Let us now \textbf{bound the scaling limit in~\eqref{2.3} from above}. For this, observe that, due to Remark~\ref{rem:denom}, the indifference price for $f(S_T)$ behaves for exploding risk aversion $\lambda=A/H$ like the certainty equivalent
\begin{align*}
    c_H:=\frac{1}{A/H} \log \inf_{\gamma} \E_{\P}\left[\exp\left(-\frac{A}{H}(V^\gamma_T-f(S_T))\right)\right].
\end{align*}
By standard density arguments, $c_H$ is continuous in $H>0$. When taking the scaling limit $H \downarrow 0$, we can thus restrict attention to $H>0$ such that $T/H \in \{1,2,\dots\}$.
If we confine ourselves to $(\cG^H_t)$-predictable strategies $\gamma$ that only intervene at times in $\TT_{H/M}$ as defined in the previous lemma, the above certainty equivalent can only grow and we can use the discrete-time duality Theorem~\ref{thm:duality} to estimate it by
\begin{align*}
    c_H \leq \E_{\QQ_{H/M}}\left[f(S_T)-\frac{1}{A/H} \log \frac{d\QQ_{H/M}}{d\P}\right]
\end{align*}
where $\QQ_{H/M} \sim \P$ is a probability that exhibits the relaxed martingale property~\eqref{eq:51} for $\TT=\TT_{H/M}$ and that has finite entropy relative to $\P$. This entropy can be bounded from below by  Lemma~\ref{lem:entropybound} and we thus obtain
\begin{align}\label{referee}
    c_H \leq \E_{\QQ_{H/M}}\left[f(S_T)-\frac{1}{A/H} \frac{M}{2H\sigma^2(M+1)}\left(S_{T}-s-\sigma W^{\QQ_{H/M}}_{T}\right)^2\right]
\end{align}
for the $\QQ_{H/M}$-Brownian motion $W^{\QQ_{H/M}}$ induced by $W$ via Girsanov's theorem; here we used that $T=T_{H/M} = \max \TT_{H/M}$ which holds because $T/H \in \{1,2,\dots\}$ by our assumption on $H$.

Now, if $S_T$ is measurable with respect to $\sigma(W^{\QQ_{H/M}}_u, \; u \in [0,T])$, we can apply the martingale representation theorem to find a predictable $\nu \in L^2(\P \otimes du)$ with $f(S_T) = s + \int_0^T \nu_u dW^{\QQ_{H/M}}_u$; an upper bound in the form required by~\eqref{2.3} emerges then from~\eqref{referee} by use of It\^{o}'s isometry and letting $M \uparrow \infty$.
In general, though, $S_T$ is just $\cF^W_T$-measurable. Following Theorem~1 in~\cite{S:76}, we exploit that our probability space $(\Omega,\cF,\QQ_{H/M})$ is rich enough to support a continuously distributed random variable independent of $W^{\QQ_{H/M}}_T$  such as  $U:=W^{\QQ_{H/M}}_T-2W^{\QQ_{H/M}}_{T/2}$ to find a Borel-measurable function $\mathbf{s}:\RR^2 \to \RR$ for which
\begin{align*}
    \Law\left((S_T,W^{\QQ_{H/M}}_T)\;\middle|\;\QQ_{H/M}\right) = \Law\left((\mathbf{s}(U,W^{\QQ_{H/M}}_T),W^{\QQ_{H/M}}_T)\;\middle|\;\QQ_{H/M}\right).
\end{align*}
This allows us to continue our estimate (\ref{referee}) as follows:
\begin{align*}
    c_H &\leq \E_{\QQ_{H/M}}\left[f(\mathbf{s}(U,W^{\QQ_{H/M}}_T))- \frac{M}{2A\sigma^2(M+1)}\left(\mathbf{s}(U,W^{\QQ_{H/M}}_T)-s-\sigma W^{\QQ_{H/M}}_{T}\right)^2\right]
    \\&=\E_{\P}\left[f(\mathbf{s}(U',W_T))- \frac{M}{2A\sigma^2(M+1)}\left(\mathbf{s}(U',W_T)-s-\sigma W_{T}\right)^2\right]
\end{align*}
where $U':=W_T-2W_{T/2}$ is independent of $W_T$ under $\P$ with the same distribution as $U$ under $\QQ_{H/M}$. Clearly, the fact that $S_T$ is square-integrable under $\QQ_{H/M}$ with expectation $s$, translates into $\mathbf{s}(U',W_T)$ having the same properties under $\P$. So, by It\^{o}'s representation theorem it is of the form $s+\int_0^T \nu_u dW_u$ for some $(\cF^W_t)$-predictable $\nu \in L^2(\P\otimes dt)$ and we obtain that
\begin{align*}
    c_H &\leq \E_{\P}\left[f\left(s+\int_0^T \nu_t dW_t\right)- \frac{M}{2A\sigma^2(M+1)}\left(\int_0^T (\nu_t-\sigma) dW_t\right)^2\right].
\end{align*}
Letting $M\uparrow \infty$ and using It\^{o}'s isometry yields the desired upper bound.