text:
Chapter 150: Absolute Predator (3)
Translation: Tay
You halfwit?!
Luciferre's eyes shook.
In his entire existence, he had never been this humiliated.
Even when he tried to become a god and failed, everyone was afraid of his strength.
But this guy, who the hell was he?
This place was a seal and a spirit world.
In this place, Luciferre was able to use his true strength.
Of course, Muyoung would have also felt his strength that reached a demigod's level.
However, Muyoung didn't even blink his eyes.
As if he dared him to act.
"Do you want to die?"
Luciferre spoke.
However, Muyoung just made a smile.
'He's no different than a scared dog.'
The more frightened a dog, the louder the bark.
Luciferre. Why would you make threats you won't be able to keep?
This is why Muyoung said he was no different than a child.
From when he was born, Luciferre was strong and had tremendous divine powers in his hands.
He lived his life with no worries and troubles.
Due to this, no matter how long he had lived, he wasn't as good as Muyoung who lived a single century.
"Keep trying to endure it, halfwit. If you're not going to give me control, I can just steal it from you."
"You say you're going to steal it from me? Ha!"
Lucifierre laughed at Muyoung.
He only suffered before because he didn't know Muyoung's soul was invading his, but since he now knew, he was able to fully defend himself.
How could a savage human invade his soul any further, he who had reached demigod level?
He was bluffing. It was obvious that Muyoung was in a hurry.
Muyoung's soul would be mixed together with his huge soul and it would soon not even have a form.
"If you can't stop it even after you know, you're really a halfwit, a fool."
Muyoung didn't care and once again exited into the darkness.
Muyoung was a great assassin.
He was the greatest assassin, one who killed Wung Chunglin and became number 0.
Even if the opponent knew of the impending assassination, Muyoung was the type to still succeed.
There was only a difference in time, and he had never failed.
Muyoung was able to obtain four different classes.
He had the potential to do so.
It was because his soul was that vast.
Class. The strength related to it was placed in the soul.
Normally, they say if you obtained one type of class, your soul was used up completely.
If you considered it simply, Muyoung's threshold was four times the norm.
Even more, if you consider the fact that he could combine other tendencies, he was an owner of a vast soul that surpassed simple calculations.
However, Luciferre didn't know this truth.
A human's soul desired a demigod's soul?
Ofcourse a human wouldn't be able to withstand it. It would destroy itself.
That was common sense.
However, Muyoung was a being outside of common sense.
It didn't take long for him to realize that his thought of self-destruction was a mistake.
"How dare a mere human?!!"
Muyoung was darkness.
He also mastered hiding his body within his mind's territory.
How in the world?
A spirit, a soul didn't have a form from the beginning.
It didn't make sense to hide it.
But, he had done it.
He assimilated into his surroundings and when there was an opportunity, he would extract some of Luciferre's soul.
The amount was very little but it was extremely unpleasant.
'You're more used to managing souls than me, the Evil Spirit Sovereign?!'
It wasn't possible.
Although the soul didn't have a form, it had a basic color.
The life of the person. The color changed depending on his ups and downs.
Even if he wanted to hide it, it wasn't possible due to this.
If he was to hide it, it would be because his soul was white.
However, a person with a white soul couldn't exist in this world.
But, Muyoung's soul had different kinds of color.
If he combined them all, it became an opaque white. It was absurd.
'His protective color…'
Luciferre thought of a chameleon.
An animal evolved to assimilate to its surrounding color to protect itself.
He was proficient enough for it be like a natural ability.
Luciferre's thoughts were mostly correct.
Muyoung had evolved like that to instinctively protect his character and his soul.
He protected his identity by looking at other people's stories after taking their Status Viewers.
On the other hand, it meant Muyoung couldn't remember who he was exactly.
It was the reason why he couldn't help but have different colors.
On the contrary, it was the reason he could become a white color.
"Are you afraid?"
He suddenly appeared.
From behind, he whispered quietly in his ears.
When he turned around, Muyoung wasn't there.
Luciferre searched within the darkness.
Of course, he was unable to find Muyoung.
"Come out! You stupid rat!"
The spirit world greatly shook.
However, there was a very small amount of 'insecurity' mixed within his voice.
To Luciferre, Muyoung was the unknown.
There was no one like him until now.
Even when he fought against numerous specters, he had never seen someone like him before.
The unknown was fear.
Luciferre was currently feeling fear.
Because he didn't want to admit it, he struggled more.
His bloated body was easier to become a target.
A boundless vicious cycle.
Luciferre's last hope was for Muyoung to self-destruct as he tried to take a demigod's soul.
However, he didn't create any cracks at all.
He absorbed whatever he took in.
What kind of a person could handle a demigod's soul?
Was he… truly a human?
Luciferre couldn't catch him. He couldn't find him.
He couldn't see him or feel him.
Like a ghost. No, not a ghost.
Something unknown.
A mysterious existence.
It was unthinkable for him to feel this sort of emotion towards a human.
"There is no color. No scent. No form…"
"I'm not your delusion."
He approached.
However, Luciferre was not able to budge.
He was caught. His thoughts were read.
Luciferre tried to treat the unknown existence of Muyoung as his delusion.
It was because if this was all a product of his imagination, everything was explainable.
What was scarier was the fact that Muyoung didn't demand it.
He just ate up little by little off Luciferre's soul endlessly.
If a few decades or centuries were to pass by like this, Luciferre's soul would disappear without a trace.
To humans, a hundred years was a long time but to Luciferre, a hundred years was like an instant.
Due to the difference in perception, it made Luciferre more frightened.
Not an entire 100 years, but only a 100 years.
To humans, it was like knowing you'd die in a year.
'Control…'
He just needed to hand over control of the body.
Afterwards, the soul controlling the body couldn't consume Luciferre's.
However, would he be able to escape this place just because he simply gave him control?
Muyoung's strength was like a drop in the ocean compared to Luciferre's.
If Luciferre wasn't able to get out, Muyoung also couldn't.
"Do you have a way to get out of this place?"
"Of course I do. Unlike your empty head, I'm one who thinks a lot."
All of his words were insulting.
Muyoung was insulting Luciferre.
He had never been insulted this much in his life before.
"What is that method?"
"I'll tell you if you hand over control."
"…Alright. However, you also need to give me something."
"Tell me."
"Your identity! Tell me your true identity."
Luciferre couldn't accept it.
The fear he felt.
He found that his fear was based on the unknown.
So, if he could just figure out Muyoung's identity, this fear would also disappear.
Muyoung grinned.
His true identity.
There was one thing.
"The absolute."
There was no distinct characteristic and identity like this.
At the same time, Muyoung's mental landscape was filled with red light.
Like the Sky Devil's, but it wasn't a disgusting light, rather it felt sacred.
What he said was funny but he couldn't laugh.
'He's chosen by the star?'
Even more, the Star of Absolute had chosen him.
Luciferre was taken back. However, Muyoung was quite far from being an absolute.
Nevertheless, he didn't think it was false.
It might be that he was rationalizing.
That because he was the chosen one of the Star of the Absolute that he could manage his soul.
He detested playing this hide and seek game any further.
Luciferre bit his lips and declared his defeat.
Luciferre became small once again.
It seemed like he didn't think the superiority of his soul would fall behind.
It was because Muyoung's elusiveness that even hid his soul made Luciferre sick.
If it was a physical fight, there was no way for Luciferre to lose against Muyoung.
"Then, I'll get out."
As if his victory was obvious, he moved freely.
Muyoung was reckless.
It seemed like he studied quite a lot about the Great City.
He probably thought that the Ancient King's castle was a good place to summon the Sky Devil.
However, Muyoung knew a lot more about this castle.
There would probably be no one else who knew the Ancient King's castle better than Muyoung.
'This place is a graveyard. So, there must be a gravekeeper.'
A gravekeeper managed graves.
Of course, they could also remove the seal.
And the gravekeeper of the Great City was the Sun Guild.
It was because it was the Sun Guild who unlocked the seal of the Ancient King, captured him, and built this Great City.
The Sun Seal!
That was the key.
Muyoung had the seal.
Bae Sungmin had fallen.
Tacan and Calla were also unable to stop Moon.
A slight difference.
Because Moon had also lost a lot of strength, he wasn't in the position to be relaxed.
"You wretched bastards. Die and become nutrients for the Sky Devil."
They fought for three long days.
Probably because they were already dead, they didn't die.
Nor did they did get tired.
An endless war of attrition.
To end this seemingly endless fight, Moon raised his lute.
Zzzzzing!
At that moment, the black sphere split.
When a sword popped out and split the sphere, Muyoung jumped out from that place.
"Ho-how did you?"
Moon widened his eyes.
That seal was his critical move.
It was even strong enough to seal Luciferre as well.
But, was there a way to escape the seal?
Even if there was one, how could Luciferre know about it?
However, when he looked carefully, it seemed like he changed a bit.
He wasn't Luciferre.
"Hellhorse. Unicorn."
Neiiighhh!
Soon, two horses approached from the sky while they spread their wings wide.
The black Hellhorse.
The pure white Unicorn.
The two only followed Muyoung.
They were able to see the nature of the person and when Luciferre came out, they hid themselves.
However, since the real Muyoung came out, there was no reason for them to stay put.
Moon spoke as the Hellhorse and the Unicorn stood beside Muyoung.
Yes, it was Muyoung.
Not Luciferre!
After realizing this, Moon lightly trembled.
A mere human drove away a demigod like being?
How could a mere human do that?!
If he had really driven away Luciferre, it was no small matter.
There was no way Moon wouldn't know the true meaning of it.
A completely unexpected move.
Even more, a move that he didn't make any preparations for had appeared before him.
"Are you afraid of me?"
Muyoung spoke.
It was the same question he asked Luciferre.
It was so.
To Moon, Muyoung had once again become the unknown.
The unknown would produce fear and Muyoung knew about this better than anyone else.
He also knew how to use it.
It was no different than resuscitation.
A crisis had become an opportunity.
On the other hand, Moon was facing the backlash of his own trick.
Because if he hadn't placed that sort of trap on Tacan, Muyoung wouldn't have gained this sort of strength.
Luciferre wouldn't have appeared and he might have even lost the fight against Moon.
Muyoung's eyes sank bleakly.
Fragments of the Fissure.
And the Authority Moon had.
'I'll eat it all up.'


text:
\section{Discussion}

\section{Conclusions and Future Work}
\label{sec:conclu}

In this paper, we propose the Siamese Multi-depth Transformer-based Hierarchical (SMITH) Encoder for document representation learning and matching, which contains several novel design choices like two level hierarchical Transformers to adapt self-attention models for long text inputs. For model pre-training, we propose the masked sentence block language modeling task in addition to the original masked word language modeling task in BERT, to capture sentence block relations within a document. The experimental results on several benchmark datasets show that our proposed SMITH model outperforms previous state-of-the-art Siamese matching models including HAN, SMASH and BERT for long-form document matching. Moreover, our proposed model increases the maximum input text length from 512 to 2048 when compared with BERT-based baseline methods.

As a part of this work, we plan to release a large scale benchmark collection for the document matching task so that it is easier for researchers to compare different document matching methods in the future. It is also interesting to investigate how to utilize the learned document representation from Transformer-based hierarchical encoders for other document-level language understanding tasks like document classification, clustering and ranking

\section{Experiments}
\label{sec:exps}
\subsection{Dataset Description}
\label{sec:data_desc}

\begin{table}[t]
    \small
	\centering
	\caption{The statistics of experimental datasets. \# of DocPairs denotes the number of document pairs. AvgSPerD, AvgWPerD, AvgWPerS denote the average number of sentences per document, average number of words per document and average number of words per sentence respectively.}
	\vspace{-0.1in}
	\label{tab:exp_data_stat_train_valid_test}
	\begin{tabular}{l|l|l|l|l|l|l}
		\hline  \hline
		Data & \multicolumn{3}{c|}{Wiki65K}          & \multicolumn{3}{c}{AAN104K}          \\ \hline
		Items                       & Train       & Valid       & Test       & Train       & Valid       & Test       \\ \hline
		\# of DocPairs              & 65,948      & 8,166      & 8,130      & 104,371     & 12,818     & 12,696     \\ \hline

		AvgSPerD    & 92.4       & 92.0      & 91.0      & 111.6      & 111.4     & 111.1     \\ \hline

		AvgWPerD      & 2035.3     & 2041.7    & 1992.3    & 3270.1     & 3251.2    & 3265.9    \\ \hline
		AvgWPerS  & 22.0       & 22.2      & 21.9      & 29.3          & 29.2         & 29.4         \\ \hline \hline
	\end{tabular}
\end{table}

Following \cite{10.1145/3308558.3313707}, we evaluate our proposed methods with two datasets: Wikipedia relevant document recommendation data (Wiki65K) and ACL Anthology Network paper citation suggestion data (AAN104K) from the previous related work. The statistics of the experimental datasets are shown in Table \ref{tab:exp_data_stat_train_valid_test}. Note that we do not use any TREC datasets like TREC 2019 Deep Learning Track data~\cite{craswell2020overview}. This is because these datasets focus on short-to-long matching like document ranking, aiming to match a short query to a set of documents, whereas our task is more on long-to-long document matching.

\subsubsection{\textbf{Relevant Document Recommendation Data}}
Relevant document recommendation can be useful in many real word applications such as news recommendation, related Web page recommendation, related QA posts recommendation, etc. We use the Wikipedia relevant document recommendation data from Jiang et. al. \cite{10.1145/3308558.3313707} as the evaluation set. The ground truth of document similarity is constructed based on the Jaccard similarity between the outgoing links of two Wikipedia documents with the assumption that similar documents have similar sets of outgoing links. The document pairs with similarities greater than 0.5 are considered as positive examples. For each positive document pair, the document with the lexicographical smaller URL is defined as the source document of the pair. Then a mismatched document from the outgoing links of the source document is sampled to generate a negative document pair. This negative sampling approach is better than random sampling from the entire corpus, since random sampled documents may be too irrelevant to make the task challenging enough to evaluate the performance of different methods. For more details of this dataset, we refer interested readers to  \cite{10.1145/3308558.3313707}.

Note that there are around six thousands training examples and less than one thousand validation/testing examples in the Wikipedia data used in  \cite{10.1145/3308558.3313707} \footnote{\url{https://research.google/pubs/pub47856/}}. We produce a Wikipedia document matching dataset of ten times larger size as shown in the data statistics in Table \ref{tab:exp_data_stat_train_valid_test}. Thus the training/validation/testing data partition and statistics are different from the data in  \cite{10.1145/3308558.3313707} and we report the results from running the model implementation on these datasets, which are different from the numbers reported in \cite{10.1145/3308558.3313707}.  We refer to this data as Wiki65K since there are 65K training document pairs.

\subsubsection{\textbf{Paper Citation Suggestion Data}}
Paper citation suggestion can help researchers find related works and finish paper writing in a more efficient way. Given the content of a research paper and the other paper as a candidate citation, we would like to predict whether the candidate should be cited by the paper. We use the ACL Anthology Networks (AAN) Corpus \cite{10.5555/1699750.1699759} processed by Jiang et. al. \cite{10.1145/3308558.3313707}  for the evaluation. The AAN data consists of 23,766 papers written by 18,862 authors in 373 venues related to NLP. For each paper with available text, the paper with each of its cited paper in the corpus is treated as a positive example. For each positive example, an irrelevant paper is randomly sampled to generate a negative example. The reference sections were removed to prevent the leakage of ground truth. The abstract sections were also removed to increase the difficulty of the task. We also filter document pairs with no section content or invalid UTF-8 text based on the processed data in \cite{10.1145/3308558.3313707} .  We refer to this data as AAN104K since there are 104K training document pairs.

\subsubsection{\textbf{Unsupervised Language Model Pre-training Data}}
\label{sec:pretrain-data}

For the SMITH model pre-training, we create a randomly sampled Wikipedia collection, containing 714,800 documents with 43,532,832 sentences and 956,169,485 words. We pre-train SMITH model with unsupervised masked word and masked sentence block language modeling loss on this data and fine-tune the model on Wiki65K and AAN104K for the downstream document matching task.

\begin{table*}[t]
    \small
	\centering
	\caption{Comparison of different models over Wiki65K and AAN104K datasets. The best performance is highlighted in boldface. SMITH-WP+SP shows significant improvements over all baseline methods with p < 0.05 measured by Student's  t-test. Note that SMITH-Short with input documents with maximum length larger than 512 and MatchBERT with input documents with maximum length larger than 256 will trigger the out-of-memory (OOM) issues on TPU V3. There is $\times 2$ in the BestDocLen, which denotes the best setting of the maximum input document length, since all the compared models are ``Dual-Encoder''/ ``Siamese'' models. Note that all the compared models are for the document/ document matching task. Models designed for the query/ document matching task are not comparable.}
	\vspace{-0.1in}
	\label{tab:main_eval_results}
	\begin{tabular}{l|l|l|l|l|l|l|l|l|l}
		\hline  \hline
		      &  Data  & \multicolumn{4}{c|}{Wiki65K}           & \multicolumn{4}{c}{AAN104K}           \\ \hline
		Method     & BestDocLen  & Accuracy & Precision & Recall & F1     & Accuracy & Precision & Recall & F1     \\ \hline \hline
		HAN (NAACL16)& 2048 $\times$ 2 & 0.8875   & 0.8571    & 0.9317 & 0.8928 & 0.8219	& 0.7895 &	0.8654 & 0.8257\\ \hline
		SMASH (WWW19)& 2048 $\times$ 2 & 0.9187   & 0.9177    & 0.9177 & 0.9177 & 0.8375	& 0.8224 & 0.8333 & 0.8278 \\ \hline
		MatchBERT  & 256 $\times$ 2 & 0.9316   & 0.9272    & 0.9366 & 0.9319 & 0.8355   & 0.8387    & 0.8201 & 0.8293 \\ \hline \hline
		SMITH-Short & 512 $\times$ 2 & 0.9415   & 0.9178    & 0.9699 & 0.9431 & 0.8212   & 0.8169    & 0.8161 & 0.8165 \\ \hline
		SMITH-NP   & 1536 $\times$ 2 & 0.9054	& 0.8911	& 0.9237	& 0.9071	& 0.7725	& 0.8106	& 0.7062	& 0.7548 \\ \hline
		SMITH-WP   &  1536 $\times$ 2   & 0.9492	& 0.9307    & 0.9707 &	0.9503 & 	0.8400 & 	0.8408 & 	0.8354 &	0.8381 \\ \hline
		SMITH-WP+SP &  1536 $\times$ 2  & \textbf{0.9585}$^\ddagger$	& \textbf{0.9466}$^\ddagger$	 & \textbf{0.9720}$^\ddagger$ &	\textbf{0.9591}$^\ddagger$ &	\textbf{0.8536}$^\ddagger$ &	\textbf{0.8431}$^\ddagger$ &	\textbf{0.8657}$^\ddagger$ &	\textbf{0.8543}$^\ddagger$ \\ \hline
		$\Delta$ over SMASH  & NA & +4.33\%	& +3.15\%	& +5.92\% & +4.51\%	& +1.92\%	& +2.52\%	& +3.89\%	& +3.19\% \\ \hline
		$\Delta$ over MatchBERT & NA  & +2.89\%	& +2.09\%	& +3.78\%	& +2.92\%	& +2.17\%	& +0.52\%	& +5.56\%	& +3.01\% \\ \hline \hline
	\end{tabular}
\end{table*}

\subsection{Experimental Setup}

\subsubsection{\textbf{Competing Methods.}}
We consider different types of methods for comparison as follows:
1) \textbf{Hierarchical Attention Network (HAN).} The HAN model \cite{yang-etal-2016-hierarchical} is a hierarchical attention model to learn document representations. For each sentence in the document, it applied an attention-based RNN to learn the sentence representation and to attend differentially to more or less important document content.
2) \textbf{SMASH.} The SMASH model \cite{10.1145/3308558.3313707} is the state-of-the-art model for long document matching. It adopts a Siamese multi-depth attention-based hierarchical recurrent neural network (SMASH RNN) to learn long document representations for matching.
3) \textbf{MatchBERT.} The MatchBERT model has been presented in Section \ref{sec:method_match_bert}.
4) \textbf{SMITH.}  This is our proposed method. We fixed the document level Transformers as 3 layers and tried several SMITH model variations as follows:

\begin{itemize}
    \item \textbf{SMITH-Short}: the SMITH model with loading the pre-trained BERT checkpoint released by Devlin et al. \cite{devlin2018bert}. We load the BERT-Base checkpoint pre-trained on uncased text and then fine-tune the model with only the document matching loss. The maximum input text length is 512 (4 sentence blocks with 128 tokens per sentence block).
    \item \textbf{SMITH-NP}: the SMITH model without language modeling pre-training stage and trained from randomly initialized model parameters. We only train SMITH-NP model with the document matching data using text matching loss.
    \item \textbf{SMITH-WP}: the SMITH model pre-trained with masked word prediction loss in the pre-training collection and then fine-tuned with document matching loss on the downstream matching task data.
    \item \textbf{SMITH-WP+SP}: the SMITH model pre-trained with both masked word prediction loss and masked sentence block prediction loss on the pre-training collection and then fine-tuned with document matching loss on the downstream matching task data.
\end{itemize}

Note that we do not compare with any interaction-focused models like DRMM~\cite{Guo:2016:DRM:2983323.2983769}, K-NRM~\cite{Xiong:2017:ENA:3077136.3080809}, Duet~\cite{Mitra:2017:LMU:3038912.3052579} or MatchPyramid~\cite{DBLP:conf/aaai/PangLGXWC16}. These models either do not scale to long documents or require heavy interactions between word pairs in two text sequences, which will lead to long inference latency in practice. Thus all the compared methods belong to representation-focused models or ``Dual-Encoder'' models where the document representation can be learned offline in advance before serving online with fast similarity search libraries \cite{JDH17,NIPS2017_7157}. Models like DRMM, KNRM, Duet, etc. are proposed for short-to-long text matching like query/document matching instead of long-to-long text matching that we focus on in this paper.

\subsubsection{\textbf{Evaluation Methodology}.}
We formalize the document matching task as a classification task where we would like to predict whether two documents are relevant or not given a document pair. Thus we consider standard classification metrics including accuracy, precision, recall and F1-score for the evaluation.

\subsubsection{\textbf{Parameter Settings and Implementation Details}}
All models are implemented with TensorFlow\footnote{\url{https://www.tensorflow.org/}}. We use TPU V3\footnote{https://cloud.google.com/tpu/} for the model pre-training and fine-tuning. For the model pre-training stage, we pre-train SMITH on the Wikipedia collection presented in Section \ref{sec:pretrain-data} around 68 epochs until the validation loss does not decrease significantly. Pre-training of SMITH with 2 layers in the sentence level Transformers and 3 layers in the document level Transformers with max document length 1024 takes 50 minutes for one epoch on the Wikipedia collection and the pre-training stage takes around 57 hours. The pre-training loss depends on the model variation types (masked word prediction loss for SMITH-WP or the sum of masked word prediction loss and masked sentence block prediction loss for SMITH-WP+SP). We dynamically mask 2 sentence blocks per document if we use the masked sentence block prediction loss presented in Section \ref{sec:smith_model_pre_training} during pre-training. The masked word prediction task follows the similar way by Devlin et al. \cite{devlin2018bert}. The sentence block length is 32. We tune max number of sentence blocks with values in $\{32, 48, 64\}$. Thus the max document length can be values in $\{1024, 1536, 2048\}$. We finally set the number of sentence blocks as $48$ (max document length is $1536$) for both Wiki65K and AAN104K, which achieves the best performance on the validation data. We tune parameters using the validation dataset and report model performance on the test dataset.
Let $L_1, H_1, A_1$  and $L_2, H_2, A_2$ denote the number of Transformer layers, the hidden size, the number of attention heads in sentence level Transformers and document level Transformers. We fix $L_2 = 3$ and tune $L_1$ with values in $\{2, 4, 6\}$. We finally set $L_1=6$, $H_2=256, A_2=4$ and $H_1=256, A_1=4$ for both Wiki65K and AAN104K. Both training and evaluation batch size are 32.  We optimize the models using Adam with learning rate 5e-5, $\beta_1=0.9, \beta_2=0.999, \epsilon = 1e-6$. The dropout rate in all layers is 0.1.

For the model fine-tuning stage, the hyper-parameters are almost the same to those used in the pre-training stage. The max number of training steps is 100K. The number of learning rate warm up steps is 10K. We remove both masked word prediction loss and masked sentence block prediction loss during fine-tuning, and update the pre-trained model parameters only using the document matching loss. The fine-tuning stage takes much less time ranging from 4 to 12 hours depending on the model and data settings.

\subsection{Evaluation Results}

We present evaluation results of different models over Wiki65K and AAN104K data  in Table \ref{tab:main_eval_results}. We summarize our observations as follows: 1) Both SMITH-WP and SMITH-WP+SP models outperform all the baseline methods including the stage-of-the-art long text matching method SMASH and MatchBERT based on pre-trained BERT models on both Wiki65K and AAN104K consistently. The comparison between SMITH-WP/ SMITH-Short and MatchBERT shows the effectiveness of introducing hierarchical document structure modeling with sentence level and document level Transformers for long document representation learning and matching. 2) If we compare the SMITH model settings with the pre-training stage (SMITH-Short, SMITH-WP, SMITH-WP+SP) with the SMITH model settings without the pre-training stage (SMITH-NP), we can find that language modeling pre-training can help increase the performance of the downstream document matching task by a large margin. Thus better language understanding via large scale language modeling pre-training will lead to better downstream task performance, which is consistent with the findings by Devlin et al. \cite{devlin2018bert}. 3) Both SMITH-WP and SMITH-WP+SP outperform SMITH-Short, which is initialized by the pre-trained open source BERT model. We think the main reason is that currently SMITH-Short can only process at most $512$ tokens due to TPU memory issues, which will hurt the performance. On the other hand, Both SMITH-WP and SMITH-WP+SP can process as long as $2048$ tokens, which is a better setting for long document representation learning. 4) If we compare SMITH-WP with SMITH-WP+SP, we can find that adding the masked sentence block prediction task presented in Section \ref{sec:smith_model_pre_training} during the pre-training stage can also be helpful to improve the downstream document matching performance. The masked word prediction task proposed  by Devlin et al. \cite{devlin2018bert} can capture the word relations and dependencies in the pre-training corpus, whereas the masked sentence block prediction task can additionally force the model to learn the sentence-level relations and dependencies. Thus combining the masked word prediction task and the masked sentence block prediction task can contribute to a better pre-training language model for long document content understanding and better downstream document matching task performance.

\subsection{Impact of Document Length}

We further analyze the impact of document length on the document matching performance. We fix the number of layers in the sentence level and the document level Transformers as 4 and 3, the max sentence block length as 32. Then we vary the number of looped sentence blocks per document for SMITH-WP+SP on Wiki65K and AAN104K with different values from 2 to 64. Thus the maximum document length increases from 64 to 2048. The performances of SMITH-WP+SP with different choices of maximum document length is shown in Figure \ref{fig:tune_doc_len}. We can find that in general SMITH-WP+SP will achieve better performance as the maximum document length increases. This confirms the necessity of long text content modeling for document matching. The SMITH model which enjoys longer input text lengths compared with other standard self-attention models is a better choice for long document representation learning and matching. We also studied the impact of different sentence block lengths and found it has no major impact on the final performance.

\begin{figure}[t]
	\centering
	\begin{subfigure}[b]{0.24\textwidth}
		\includegraphics[width=\textwidth]{figures/doc-len-wiki65k}
		\label{fig:doc-len-wiki}
	\end{subfigure}
	\hspace{-0.15in}
	~

	\begin{subfigure}[b]{0.24\textwidth}
		\includegraphics[width=\textwidth]{figures/doc-len-aan104k}
		\label{fig:doc-len-aan}
	\end{subfigure}
	\vspace{-0.4in}
	\caption{Performance of SMITH-WP+SP on the validation data with different choices of maximum document length.}\label{fig:tune_doc_len}
\end{figure}

\subsection{Impact of the Number of Layers in Sentence Level Transformers}
Next we analyze the impact of the number of layers in the sentence level Transformers on the final document matching performances. We set the document level Transformer layers as 3, the maximum sentence block length as 32, the number of sentence blocks per document as 48. So the maximum document length is 1536. Then we vary the number of layers in the sentence level Transformers and observe the change of the performances of SMITH-WP+SP. The results are shown in Table \ref{tab:impact_sent_level_transformer_layers}. We can find that the setting with 4 or 6 layers in the sentence level Transformer layers is slightly better than the setting with only 2 layers in the sentence level Transformers. Increasing the layers in the sentence level Transformers can help the model to learn sentence block semantic representation with more high level interactions. However, it also leads to larger memory cost such as the intermediate activation in each layer. Thus in practice this hyper-parameter has to be tuned with the validation dataset for the trade-off between the model capacity and memory cost.

\begin{table}[t]
	\small
	\centering
	\caption{The document matching performance  with different choices of the number of layers in the sentence level Transformers on the validation data. $L_1$ denotes the number of layers in the sentence level Transformers. }
	\vspace{-0.3cm}
	\label{tab:impact_sent_level_transformer_layers}
	\begin{tabular}{l|l|l|l|l|l}
		\hline \hline
		Data                     & $L_1$ & Accuracy & Precision & Recall & F1     \\ \hline \hline
		\multirow{3}{*}{Wiki65K} & 2    & 0.9537   & 0.9449    & 0.9635 & 0.9541 \\ \cline{2-6}
		& 4    & 0.9589   & \textbf{0.9479}    & 0.9718 & 0.9597 \\ \cline{2-6}
		& 6    & \textbf{0.9594}   & 0.9426   & \textbf{0.9784} & \textbf{0.9602} \\ \hline
		\multirow{3}{*}{AAN104K} & 2    & 0.8566   & 0.8470    & 0.8612 & 0.8540 \\ \cline{2-6}
		& 4    & 0.8573   & 0.8426    & \textbf{0.8694} &  \textbf{0.8558} \\ \cline{2-6}
		& 6    & \textbf{0.8580}   & \textbf{0.8508}    & 0.8591 & 0.8549 \\ \hline \hline
	\end{tabular}
\end{table}

\subsection{Impact of Combining Representations from Different Levels}

\begin{table}[t]
\small
	\centering
	\caption{The document matching performance  with different choices of the document representation combining methods presented in Section \ref{sec:doc_combine_representation_from_different_levels} on the validation data.}
	\vspace{-0.3cm}
	\label{tab:impact_different_doc_combine_mode}
\begin{tabular}{l|l|l|l|l|l}
\hline \hline
Data                     & Combine & Accuracy & Precision & Recall & F1     \\ \hline
\multirow{4}{*}{Wiki65K} & Normal        & \textbf{0.9594}   & \textbf{0.9426}    & 0.9784 & \textbf{0.9602} \\ \cline{2-6}
                         & Sum-Concat   & 0.9192   & 0.9103    & 0.9301 & 0.9201 \\ \cline{2-6}
                         & Mean-Concat  & 0.9221   & 0.8924    & 0.9599 & 0.9249 \\ \cline{2-6}
                         & Attention     & 0.9431   & 0.9099    & \textbf{0.9836} & 0.9453 \\ \hline
\multirow{4}{*}{AAN104K} & Normal        & \textbf{0.8580}   & \textbf{0.8508}    & 0.8591 & \textbf{0.8549} \\ \cline{2-6}
                         & Sum-Concat   & 0.7632   & 0.7924    & 0.6962 & 0.7412 \\ \cline{2-6}
                         & Mean-Concat  & 0.7061   & 0.6387    & \textbf{0.9131} & 0.7516 \\ \cline{2-6}
                         & Attention     & 0.8434   & 0.8162    & 0.8758 & 0.8450 \\ \hline \hline
\end{tabular}
\end{table}

As presented in Section \ref{sec:doc_combine_representation_from_different_levels}, we evaluate the performances of SMITH-WP+SP with different methods to combine representations from different levels. Table \ref{tab:impact_different_doc_combine_mode} shows the document matching performance with different choices of document representation combing methods. We can see that the ``normal'' combing method where we only use the output of the document level Transformers as the final document representation works best. For the other three methods, the ``attention'' method is better than ``sum-concat'' and ``mean-concat''. One possible reason is that the weighted combination of sentence blocks can be helpful for generating better document representations as the attention weights can encode the relative importance of different sentence blocks on representing the document content, which is why attention based combining methods work better. The document level Transformers already learn a weighted combination based on the input sentence block representation sequences, which already provide enough signals on the importance scores of different sentence blocks in a document.

\section{Introduction}
\label{sec:intro}

Semantic matching is an essential task for many natural language processing (NLP) and information retrieval (IR) problems. Research on semantic matching can potentially benefit a large family of applications including ad-hoc retrieval, question answering and recommender systems \cite{10.5555/2683840}. Semantic matching problems can be classified into four different categories according to text length, including short-to-short matching, short-to-long matching, long-to-short matching and long-to-long matching. Table \ref{tab:different_text_match_task} shows a classification of different semantic matching tasks with example datasets. Semantic matching between short text pairs is relatively well studied in previous research on paraphrase identification \cite{DBLP:conf/naacl/YinS15}, natural language inference \cite{bowman-etal-2015-large}, answer sentence selection \cite{yang-etal-2015-wikiqa}, etc. Short-to-long semantic matching like relevance modeling between query/ document pairs has also been a popular research topic in IR and NLP communities \cite{craswell2020overview}. For long-to-short semantic matching, there are also a variety of research on tasks like conversation response ranking, which is to match a conversation context with response candidates \cite{DBLP:journals/corr/LowePSP15}. To the best of our knowledge, semantic matching between long document pairs, which has many important applications like news recommendation, related article recommendation and document clustering, is less explored and needs more research effort. Table \ref{tab:doc_match_example} shows an example of semantic matching between document pairs from  Wikipedia. These documents have thousands of words organized in sections, passages and sentences.

\begin{table*}[]
	\small
	\caption{A classification of different semantic matching tasks. The focus of this paper is \textit{long-to-long} document matching. }
	\vspace{-0.15in}
	\label{tab:different_text_match_task}
\begin{tabular}{p{1.5cm}|p{3cm}|p{2cm}|p{9cm}}
\hline \hline
Type                                                   & Tasks                           & Example Data                       & Explanation                                                                                \\ \hline
\multirow{3}{*}{Short-to-short}               & Paraphrase Identification       & MRPC \cite{dolan-brockett-2005-automatically}                          & Given two sentences, predict whether they have the same semantic meaning.                                   \\ \cline{2-4}
                                                       & Answer Sentence Selection & WikiQA \cite{yang-etal-2015-wikiqa}         & Given a question and candidate answer sentences, select a correct answer. \\ \cline{2-4}
                                                       & Textual Entailment  & SNLI \cite{bowman-etal-2015-large}                         & Given two sentences, predict whether they have textual entailment relations.             \\ \hline
\multirow{2}{*}{Short-to-long}                & Document Ranking                & TREC 2019 Deep Learning \cite{craswell2020overview} & Given a query and a candidate document set, rank documents according to query/ document relevance.        \\ \cline{2-4}
                                                       & Blog Search                    & TREC 2008 Blog Track \cite{DBLP:conf/trec/OunisMS08}
                                                                    & Given a query and a blog collection, rank blog posts according to topic relevance or opinions. \\ \hline
\multirow{1}{*}{Long-to-short}                & Response Ranking   & UDC \cite{DBLP:journals/corr/LowePSP15}           & Given a dialog context and candidate responses, select a high quality response.               \\\hline

\multirow{2}{*}{\textbf{\textit{Long-to-long}}} & Related Article Suggestion & Wikipedia \cite{10.1145/3308558.3313707}                     & Given a document pair, predict whether they are relevant with each other.                                   \\ \cline{2-4}
                                                       & Paper Citation Suggestion       & AAN \cite{10.5555/1699750.1699759}                           & Given a paper pair, predict whether one paper is a good citation of the other.               \\ \hline \hline
\end{tabular}
\end{table*}

\begin{table*}[t]
	\small
	\caption{An example to illustrate the document matching task from the Wikipedia data. ``Sim'' means the similarity estimated based on the Jaccard similarity between the outgoing links of two documents. ``Len'' means the document length by words.}
	\vspace{-0.15in}
	\label{tab:doc_match_example}
\begin{tabular}{l|p{3.5cm}|p{1.8cm}|p{7cm}|l|l|l}
\hline \hline
Type       & URL                                                    & Title                   & Document Content & Label            & Sim       & Len \\ \hline
Source & \url{http://en.wikipedia.org/wiki/Chartered\_Engineer\_(UK)} & Chartered Engineer (UK) & In the United Kingdom, a Chartered Engineer is an Engineer registered with the Engineering Council (the British ...... & \textbackslash{} & \textbackslash{} & 1147            \\ \hline
Target & \url{http://en.wikipedia.org/wiki/Engineering\_Council}      & Engineering Council     & The Engineering Council (formerly Engineering Council UK; colloquially known as EngC) is the UK's regulatory ...... & 1    & 0.5846           & 999             \\ \hline
Target & \url{http://en.wikipedia.org/wiki/Institute\_of\_Physics}    & Institute of Physics    & The Institute of Physics (IOP) is a UK-based learned society and professional body that works to advance physics ...... & 0 & 0.0099           &       2036          \\ \hline \hline
\end{tabular}
\end{table*}

Compared to semantic matching between short texts, or between short and long texts, semantic matching between long texts is a more challenging task due to a few reasons: 1) When both texts are long, matching  them requires a more thorough understanding of semantic relations including matching pattern between text fragments with long distance; 2) Long documents contain internal structure like sections, passages and sentences. For human readers, document structure usually plays a key role for content understanding. Similarly, a model also needs to take document structure information into account for better document matching performance; 3) The processing of long texts is more likely to trigger practical issues like out of TPU/GPU memories without careful model design. In the recent two years, self-attention based models like Transformers \cite{NIPS2017_Transformers} and BERT \cite{devlin2018bert} have achieved the state-of-the-art performance in several natural language understanding tasks like sentence pair classification, single sentence classification and answer span detection. These kinds of models, however, are still limited to the representation and matching of short text sequences like sentences due to the quadratic computational time and space complexity of self-attention with respect to the input sequence length \cite{NIPS2017_Transformers}. To handle this challenge, we would like to design a long-form text encoder, which combines the advantages of sequence dependency modeling with self-attention in Transformers and long text processing with hierarchical structures for document representation learning and matching.

Perhaps the closest prior research to our work is the study on the semantic text matching for long-form documents by Jiang et. al. \cite{10.1145/3308558.3313707}. They proposed the MASH RNN model to learn document representations from multiple abstraction levels of the document structure including passages, sentences and words. However, the adopted attentive RNN component in the MASH RNN model may suffer from the gradient vanishing and explosion problems on long input sequences. It is difficult for RNN based models to capture the long distance dependencies in long documents, which might lead to sub-optimal performance on long text content modeling compared with self-attention models like Transformers/ BERT where there are direct interactions between every token pair in a sequence.
It is the right time to revisit this line of work and further push the boundary of long text content understanding with self-attention models like Transformers. However, as presented before, building Transformer based long text encoder is not trivial because of the quadratic computational time and memory complexity of self-attention with respect to the input sequence length. For example, the maximum input text length of BERT is 512 for single sentence classification, and less than 512 for sentence pair classification.

We address these issues by proposing the Siamese Multi-depth Transformer-based Hierarchical (SMITH) Encoder for document representation learning and matching, which contains several novel design choices to adapt self-attention models like Transformers/ BERT for modeling long text inputs. Our proposed text matching model adopts a two-tower structure of Siamese network, where each tower is a multi-depth Transformer-based hierarchical encoder to learn the document representations. We first split the input document into several sentence blocks, which may contain one or more sentences using our proposed greedy sentence filling method. Then the sentence level Transformers learn the contextual representations for the input tokens in each sentence block. We represent the whole sentence block with the contextual representations of the first token, following the practice in BERT. Given a sequence of sentence block representation, the document level Transformers learn the contextual representation for each sentence block and the final document representation. This model design brings several benefits in terms of model training and serving: 1) The Siamese model architecture is a better choice to serve with efficient similarity search libraries for dense vectors\cite{JDH17,NIPS2017_7157}, since document representations can be generated independently and indexed offline before online serving. 2) The hierarchical model can capture the document internal structural information like sentence boundaries. 3) Compared with directly applying Transformers to the whole document, the two level hierarchical SMITH model including sentence level and document level Transformers reduces the quadratic memory and time complexity by changing the full self-attention on the whole document to several local self-attentions within each sentence block. The sentence level Transformers capture the interactions between all token pairs within a sentence block, and the document level Transformers maintain the global interaction between different sentence blocks for long distance dependencies.

Inspired by the recent success of language model pre-training methods like BERT, SMITH also adopts the ``unsupervised pre-training + fine-tuning'' paradigm for the model training. For the model pre-training, we propose the masked sentence block language modeling task in addition to the original masked word language modeling task used in BERT for long text inputs. When the input text becomes long, both relations between words in a sentence block and relations between sentence blocks within a document becomes important for content understanding. Therefore, we mask both randomly selected words and sentence blocks during model pre-training. The sum of the masked sentence block prediction loss and the masked word prediction loss is the final SMITH model pre-training loss. The model fine-tuning process is similar to BERT, where we remove the word/ sentence level masks and fine-tune the model parameters initialized with pre-trained checkpoints with only the text matching loss. We evaluate the proposed model with several benchmark data for long-form text matching \cite{10.1145/3308558.3313707}. The experimental results show that our proposed SMITH model outperforms the previous state-of-the-art Siamese matching models including hierarchical attention~\cite{Yang:2016:ARS:2983323.2983818}, multi-depth attention-based hierarchical recurrent neural network~\cite{10.1145/3308558.3313707}, and BERT for long-form document matching, and increases the maximum input text length from 512 to 2048 when compared with BERT-based baselines.

Our main contributions can be summarized as follows:

 \begin{itemize
	\item We propose the Siamese Multi-depth Transformer-based Hierarchical (SMITH) Encoder for document matching, which contains several novel design choices to adapt self-attention models for modeling long text inputs.
	\item For model pre-training, we propose  the masked sentence block language modeling task to capture sentence level semantic relations within a document towards better long text content understanding.
	\item Experimental results on several benchmark data for long-form text matching \cite{10.1145/3308558.3313707} show that our proposed SMITH model outperforms the previous state-of-the-art models and increases the maximum input text length from 512 to 2048 when comparing with BERT based baselines. We will open source a Wikipedia based benchmark dataset, code and a pre-trained model checkpoint to accelerate future research on document understanding and matching.
\end{itemize}

\section{Method Overview}
\label{sec:method_overview}

\subsection{Problem Formulation}

We define the task of document matching following previous literature \cite{10.1145/3308558.3313707}. We are given a source document $d_s$ and a set of candidate documents $\mathcal{D}_c$. The system needs to estimate the semantic similarities $\hat{y} = Sim(d_s, d_c)$, where $d_c \in \mathcal{D}_c$, for every document pair $(d_s, d_c)$ so that the target documents semantically matched to the source document $d_s$ have higher similarity scores. In practice, the documents may contain structural information like passage/ sentence/ word boundaries and different text length characteristics. The task can be formalized as a regression or classification problem depending on the type of data labels. A summary of key notations in this work is presented in Table \ref{tab:notation}.

\begin{table}[!t]
  	\footnotesize
	\caption{A summary of key notations in this work.}
	\vspace{-0.4cm}
	\begin{tabular}
		{ p{0.07\textwidth} | p{0.38\textwidth}} \hline  \hline

		$d_s, \mathcal{D}_s$ & The source document and the set of all source documents \\\hline
		$d_c, \mathcal{D}_c$ & The candidate document and the set of all candidate documents \\\hline
		$\mathbf{E}(d_s), \mathbf{E}(d_c)$ & The learned dense vector representations for $d_s$ and $d_c$ \\ \hline
		$L_1, L_2$  & The number of layers in the sentence level Transformers and in the document level Transformers\\ \hline
		$S_i, \mathbf{E}(S_i)$ & The i-th sentence block in the document and the sequence of word representations for $S_i$\\ \hline

		$W^i_j$ & The j-th word in the i-th sentence block in the document \\ \hline
		$L_d, L_s$ & The length of a document by sentence blocks and the length of a sentence block by words \\ \hline
		$t(W^i_j)$ & The token embedding for $W^i_j$ \\ \hline
		$p(W^i_j)$ & The position embedding for $W^i_j$  \\ \hline

		$\mathbf{T}^i_j, \mathbf{S}_i$ & The contextual token representation learned by sentence level Transformers for $W^i_j$ \ and the contextual sentence block representation learned by document level Trnasformers for $S_i$ \\ \hline
		$b$, $H$, $A$, $L$& The batch size, the hidden size, number of attention heads and layers in Transformers \\ \hline \hline

	\end{tabular}\label{tab:notation}
	\vspace{-10pt}
\end{table}

\subsection{Document Matching with Transformers}
\label{sec:method_transformers}

The original BERT model proposed by Devlin et. al. \cite{devlin2018bert} supports text matching as the sentence pair classification task. Two input text sequences will be concatenated and separated by a special token [SEP] to feed into the BERT model, in order to learn the contextual representation of each token. Then the contextual representation of the first token, which is the added [CLS] token, will be projected into a probability distribution estimation over different label dimensions to compute the cross-entropy loss. Directly applying this ``Single Encoder'' BERT model to the document matching task will cause two problems: 1) The input text length for each document will be very limited. On average, we can only feed at most 256 tokens per document into the BERT model to run the model fine-tuning or inference process of document matching. 2) The ``Single Encoder'' BERT model cannot be served for applications requiring high inference speed. To solve this problem, we learn query independent document representations and index them offline to serve with efficient similarity search libraries~\cite{JDH17,NIPS2017_7157}. Offline indexing of document representations requires generating dense vector representations for the two documents independently without expensive interactions in the earlier stage. This motivates us to focus on designing ``Dual Encoder'' BERT model with a Siamese network architecture, where each tower is to encode one document separately. The two towers can share model parameters. In the following sections, we introduce a basic Siamese matching model with Transformers MatchBERT (Section \ref{sec:method_match_bert}) and the Siamese hierarchical matching model SMITH (Section \ref{sec:method_smith}).
\subsubsection{\textbf{MatchBERT: A Basic Siamese Matching Model with Transformers}}
\label{sec:method_match_bert}
\begin{figure}[th]
	\center
	\includegraphics*[viewport=0mm 0mm 180mm 92mm, scale=0.32]{figures/Match-Bert-V2.pdf}
	\vspace{-0.5cm}
	\caption{The architecture of the MatchBERT model.}\label{fig:match-bert-model}
	\vspace{-0.2cm}
\end{figure}

\begin{figure*}[th]
	\center
	\includegraphics*[viewport=0mm 0mm 320mm 125mm, scale=0.50]{figures/SMITH-V3-Smith-2L.pdf}

	\vspace{-0.4cm}
	\caption{The architecture of the Multi-depth Transformer-based Hierarchical Encoder in the SMITH model for document representation learning and matching. We visualize the sentence level Transformer encoder for the 1st, 2nd and the last sentence block in a document. The output sentence representations of sentence encoders become the inputs of the document level Transformer encoder.}\label{fig:smith-model}

\end{figure*}

Figure \ref{fig:match-bert-model} shows the architecture of MatchBERT model for text matching adapted from the BERT model proposed by Devlin et. al. \cite{devlin2018bert}. There are two text encoders in MatchBERT, where each encoder is a BERT model to learn the representation of the source document $d_s$ or the candidate document $d_c$. Then we compute the cosine similarity between the pooled sequence output of two documents $cos(\mathbf{E}(d_s), \mathbf{E}(d_c))$. The text matching loss is the cross-entropy loss when we compare the document pair similarity scores with document pair labels. To handle long document input, MatchBERT will only model the first $N$ tokens of each document, where the max value of $N$ can be 512. To train the MatchBERT model, we initialize the model parameters with the open source pre-trained BERT checkpoints \footnote{\url{https://github.com/google-research/bert}} and then fine tune the model with the text matching loss. MatchBERT will be a strong baseline model.

\subsubsection{\textbf{SMITH: Siamese Hierarchical Matching Model with Transformers}}
\label{sec:method_smith}

The SMITH model, which refers to \textbf{S}iamese \textbf{M}ult\textbf{I}-depth \textbf{T}ransformer-based \textbf{H}ierarchical Encoder, is an extension of the MatchBERT model. It also adopts a Siamese network architecture, where each tower is a transformer-based hierarchical encoder to learn representations in different levels like sentence and document level of long documents. In this way, it combines the advantages of long distance dependency modeling of self-attention in Transformer encoders and hierarchical document structure modeling for long text representation learning. Figure \ref{fig:smith-model} shows the Transformer-based hierarchical encoder in the SMITH model.

\section{Siamese Hierarchical Matching Model with Transformers}

\subsection{Hierarchical Modeling for Document Representation}
\label{sec:doc_representation}

\begin{figure}[th]
	\center
	\includegraphics*[viewport=0mm 0mm 140mm 52mm, scale=0.54]{figures/greedy-sent-filling.pdf}
	\vspace{-0.4cm}
	\caption{An example of splitting a document into different sentence blocks using the greedy sentence filling method.
	Left: natural sentence splitting. Right: greedy sentence filling splitting.}\label{fig:greedy-sent-filling}

\end{figure}

\subsubsection{\textbf{Splitting Documents with Greedy Sentence Filling}}
In order to represent one document with hierarchical Transformers, we need to split the document into multiple smaller text units. A natural way to perform this step is to split a document into sentences with some off-the-shelf sentence boundary detection libraries. However, as sentence length varies a lot, padding all sentences to the same length will introduce a lot of padded tokens for short sentences, which will make the usage of the model capacity unnecessarily inefficient. We want to  preserve each sentence's semantics so methods which may break down sentences like fixed length sliding window\cite{10.1145/3331184.3331303} are not good options either. We propose a ``greedy sentence filling'' method to reduce the number of padded words and increase the actual text length the model can take as its input. Specifically, we split a document into multiple sentence blocks of predefined length so that each sentence block can contain one or more natural sentences. We try to fill as many sentences as possible into one sentence block until the block reaches the predefined max block length. When the last sentence cannot fill in the current block, we move it to the next block. When an individual sentence alone is longer than the max sentence block length, we truncate it to fit in the current block. Figure \ref{fig:greedy-sent-filling} shows an example of how we split a document into sentence blocks. We can see that greedy sentence filling greatly reduces the number of padded tokens given a fixed maximum sentence block length.

\subsubsection{\textbf{Hierarchical Document Representation Learning}}
Let $\mathcal{D}$ denote the input document. With greedy sentence filling, we split it into a sequence of sentence blocks $\{S_1, S_2, ..., S_{L_d}\}$, where $S_i = \{W^i_1, W^i_2, ..., W^i_{L_s}\}$. $S_i$ is the i-th sentence block in the document. $W^i_j$ is the j-th word in the i-th sentence block. $L_d$ and $L_s$ denote the length of a document by sentence blocks and the length of a sentence block by words respectively. We learn the representation of each sentence $S_i$ with the Transformer encoder described in \cite{NIPS2017_Transformers}, which consists of the multi-head self-attention and a position-wise fully connected feed-forward network with residual connections \cite{DBLP:journals/corr/HeZRS15}. We firstly map the words in $S_i$ to a sequence of dense vector representations

\begin{equation}\label{Eqn:word_dense_vectors}
    \small
	\mathbf{E}(S_i) = (\mathbf{e}^i_1, \mathbf{e}^i_2, ..., \mathbf{e}^i_{L_s})
\end{equation}
where $\mathbf{e}^i_j = t(W^i_j) + p(W^i_j)$ is the sum of the token embedding and position embedding of word  $W^i_j$ following the same practice in BERT. The token embedding is initialized randomly during pre-training and the position embedding follows the same setting as in Transformers \cite{NIPS2017_Transformers}. The sentence level Transformers will transform $\mathbf{E}(S_i)$ into a sequence of contextualized representations for words in the sentence block $\{\mathbf{T}^i_1, \mathbf{T}^i_2, ..., \mathbf{T}^i_{L_s} \}$. Following the setting in BERT model, we use the contextual representation of the first token, the added [CLS] token, as the learned representation of the whole sentence block. We add another dense layer and perform a L2 normalization on the sentence block representation. The final sentence block representation also adds the sentence block position embedding to model the sentence block location in the document.

With the learned sentence block representations from the sentence level Transformers and the sentence block position embeddings, the document level Transformer encoders will produce a sequence of contextual sentence block representations  $\{\mathbf{S}_1, \mathbf{S}_2, ..., \mathbf{S}_{L_d} \}$. We still use the first contextual sentence block representation  as the representation for the whole document. There will be another dense layer added to transform the document representation with L2 normalization before we compute the cosine similarity between the two document representations in the document pair $(d_s, d_c)$.

\subsubsection{\textbf{Memory and Time Complexity Analysis of Two Level Hierarchical Transformers}}
Next let's analyze memory and time complexity for the two level hierarchical Transformers. The attention mechanism used in Transformer is the scaled dot-product attention, which performs transformation from a query and a set of key-value pairs to an output.   The output representation is defined as a weighted sum of the values, where the weight to each value is computed as the interaction score between the query and the corresponding key normalized by the softmax function. Specifically, given the input query embeddings $\mathcal{Q}$, key embeddings $\mathcal{K}$ and value embeddings $\mathcal{V}$, where $\mathcal{Q} \in \mathbb{R}^{b \times l_\mathcal{Q}\times H}$, $\mathcal{K} \in \mathbb{R}^{b \times l_\mathcal{K}\times H}$, $\mathcal{V} \in \mathbb{R}^{b \times l_\mathcal{V}\times H}$, the scaled dot-product attention is defined as:

\begin{equation}\label{Eqn:transformer_scaled_dot_prod_att}
\small
\text{Attention}(\mathcal{Q}, \mathcal{K}, \mathcal{V}) = \text{softmax}\Big( \frac{\mathcal{Q}\mathcal{K}^T}{\sqrt{d}}\Big)\mathcal{V}
\end{equation}

where $l_\mathcal{Q}$, $l_\mathcal{K}$, $l_\mathcal{V}$ are the number of tokens in each sequence and $l_\mathcal{K} = l_\mathcal{V}$. $b$ is the batch size and $H$ is the hidden size. To understand the memory cost of Transformers, we can focus on the attention computation in Equation \ref{Eqn:transformer_scaled_dot_prod_att}. Let us assume  $l_\mathcal{K} = l_\mathcal{V} =  l_\mathcal{Q} = N$, then the term $\mathcal{Q}\mathcal{K}^T$ has the shape $[b, N, N]$, where $N$ is the maximum input sequence length. Let $A$ and $L$ denote the number of attention heads and layers in Transformers, then the memory complexity of the attention computation in Transformers is $O(b \cdot A \cdot N^2 \cdot L)$. This is why the memory cost of the scaled dot-product attention in Transformers grows quadratically as the increasing of the input sequence length, which makes it difficult to directly apply Transformers to very long input sequences \footnote{This is the memory cost of Transformers without considering the feed-forward layers. For the complete memory complexity analysis results including both attention and feed-forward layers in Transformers, we refer the interested readers to \cite{kitaev2020reformer}.}. Similar conclusions also hold for the time complexity of scaled dot-product used in Transformers. For two level hierarchical Transformers, let $L_s$ denote the max sentence block length by tokens. Then we will split a  document into $\frac{N}{L_s}$ sentence blocks. The memory complexity of the attention computation of sentence/ document level Transformers is

\begin{small}
\begin{eqnarray}\label{Eqn:two_level_transformer_attention_memory_cost}
&& b \cdot A \cdot {L_s}^2 \cdot L \cdot \frac{N}{L_s} + b \times A \times (\frac{N}{L_s})^2 \cdot L \\ \nonumber
&=&  ( {L_s}^2 \cdot \frac{N}{L_s} + (\frac{N}{L_s})^2 ) \cdot b \cdot A \cdot L  \\ \nonumber
&=&  (L_s \cdot N + \frac{N^2}{L_s^2}) \cdot b \cdot A \cdot L
\end{eqnarray}
\end{small}

Here we assume the number of attention heads and the number of layers are the same for the sentence level Transformers and document level Transformers for simplicity. Thus the memory complexity of two level hierarchical Transformers is $O(\frac{N^2}{L_s^2} \cdot b \cdot A \cdot L)$. Comparing with the original Transformers, we reduce the memory complexity by a factor of $L_s^2$ with only performing local self-attention over tokens in the same sentence block.

\subsubsection{\textbf{Combine Representations from Different Levels}}
\label{sec:doc_combine_representation_from_different_levels}
In order to integrate learned features from different levels of document structures, we consider several settings for generating the final document representations as follows:

\textbf{Normal}: we only use the output of the document level Transformers as the final document representation.

\textbf{Sum-Concat}: we firstly compute the sum of all sentence level representations and use the concatenation of the sum with the document level representation as the final document representation.

\textbf{Mean-Concat}: we firstly compute the mean of all sentence level representations and use the concatenation of the mean with the document level representation as the final document representation.

\textbf{Attention}: we firstly compute the weighted sum of the sentence level representations with attention mechanism: $\sum_{i=1}^{L_d} \mathbf{h}_i \cdot \text{softmax}(\mathbf{h}_i \mathbf{W} \mathbf{v})$, where $\mathbf{h}_i  \in  \mathbb{R}^{H} $ is the learned representation for the i-th sentence block by the sentence level Transformers. $\mathbf{W} \in \mathbb{R}^{H \times V}$ is the projection matrix and $\mathbf{v} \in  \mathbb{R}^{V}$ is the attention model parameter. Then we concatenate the weighted sum with the document level representation as the final document representation.

\subsection{SMITH Model Pre-training}
\label{sec:smith_model_pre_training}
For the model training of SMITH, we adopt the ``pre-training + fine-tuning'' paradigm as in BERT. This approach is to firstly pre-train the model with large unlabeled  plain text in an unsupervised learning fashion, and then fine-tune the model with a supervised downstream task so that only few parameters need to be learned from scratch. In addition to the masked word language modeling task proposed by Devlin et. al. \cite{devlin2018bert}, we also propose the masked sentence block language modeling task, because one of the basic units in the SMITH encoder for modeling documents is the sentence block. Masked sentence block prediction task can help the model learn the relations between different sentence blocks and hopefully, get a better understanding of whole documents. Our pre-training loss is a sum of masked word prediction loss and the masked sentence block prediction loss. For the details of the masked word prediction task, please refer to Devlin et. al. \cite{devlin2018bert}. For the masked sentence block prediction task, we perform dynamic sentence block masking and masked sentence block prediction as follows.

\subsubsection{\textbf{Dynamic Sentence Block Masking.}}
Let $\mathbf{D} = \{\mathbf{h}_1, \mathbf{h}_2, ..., \mathbf{h}_{L_d}\}$ denote a sequence of sentence block representations learned by the sentence level Transformers.  For each document in the current batch, we randomly sample $m$ sentence blocks $\mathcal{M} = \{\mathbf{h}_k | \mathbf{h}_k \in \mathbb{R}^H, k \in \mathcal{K}\}$ and replace these sentence blocks with a randomly initialized masked sentence block vector $\hat{\mathbf{h}} \in \mathbb{R}^H$. For example, if we randomly select the 3rd and 5th sentence block for masking, the masked document becomes $\mathbf{\hat{D}} = \{\mathbf{h}_1, \mathbf{h}_2,  \hat{\mathbf{h}}, \mathbf{h}_4, \hat{\mathbf{h}},\mathbf{h}_6, ..., \mathbf{h}_{L_d}\}$. This dynamic sampling process repeats for every document in a batch in each step, so that the same document may get different masked sentence block positions in different steps. The dynamic masking strategy enables the model to predict a larger range of sentence blocks in a document compared with the opposite static masking.

\subsubsection{\textbf{Masked Sentence Block Prediction.}}
To perform masked sentence block prediction, we consider a multi-class sentence block classification setting similar to the masked word prediction. However, we do not have a global vocabulary for different sentence blocks. \footnote{In fact, the number of all unique sentence blocks can be unlimited considering different composition of words into sentence blocks.} Instead, we collect all the masked sentence blocks in a batch as a candidate sentence block pool, from which the model will try to predict the correct sentence block. For each masked sentence block position, the original sentence block in the current position is the positive example. The other co-masked sentence blocks in the current document and in the other documents of the same batch are the negative examples. Specifically, we apply document level Transformers on the masked document  $\mathbf{\hat{D}}$ to get a sequence of contextual sentence block representations $\{\mathbf{\hat{S}}_1, \mathbf{\hat{S}}_2, ..., \mathbf{\hat{S}}_{L_d} \}$. Then $\mathbf{\hat{S}}_k$ will be used to predict the original sentence block representation $\mathbf{h}_k$. Given a batch of $B$ masked sentence blocks with the predicted sentence block representation $\mathbf{\hat{S}} \in \mathbb{R}^{B \times H}$  and the ground truth sentence block representation  $\mathbf{h} \in \mathbb{R}^{B \times H}$ where $B = b \times m$, we can compute a pairwise similarity matrix for every masked sentence block pair in the current batch as follows:

\begin{equation}\label{Eqn:similarity_matrix_batch_pairwise}
\small
\text{Sim}(\mathbf{\hat{S}}, \mathbf{h}) = \mathbf{\hat{S}}\mathbf{h}^T
\end{equation}

where $\text{Sim}(\mathbf{\hat{S}}, \mathbf{h}) \in \mathbb{R}^{B \times B}$ where $\text{Sim}(\mathbf{\hat{S}}_j, \mathbf{h}_i)$ is the predicted similarity between the j-th predicted sentence block representation and the i-th sentence block class. We normalize it with a softmax function to transform it to the predicted probability for the the i-th sentence block class as follows:

\begin{equation}\label{Eqn:predict_probability}
\small
p(\mathbf{h}_i | \mathbf{\hat{S}}_j) = \frac{\exp( \text{Sim}(\mathbf{\hat{S}}_j, \mathbf{h}_i) )}{\sum_{r=1}^{B} \exp ( \text{Sim}(\mathbf{\hat{S}}_j, \mathbf{h}_r) )}
\end{equation}

Thus all the sentence blocks $\{\mathbf{h}_r\}$, where $r \in [1, B], r \neq j$ can be treated as randomly sampled negative classes for $ \mathbf{\hat{S}}_j$. Finally we can compute the cross-entropy loss over all masked sentence blocks and the pre-training joint loss:

\begin{equation}\label{eqn:sentence-loss}
\small
\begin{aligned}
\mathcal{L}_{\text{sp}} = - \frac{1}{B}\sum_{i=1}^B\sum_{j=1}^B\mathds{1}\{j=i\} &\log{ p(\mathbf{h}_i | \mathbf{\hat{S}}_j) }
\end{aligned}
\end{equation}

\begin{equation}\label{eqn:pretrain-loss}
\small
\begin{aligned}
\mathcal{L}_{\text{pretrain}} = \mathcal{L}_{\text{sp}} + \mathcal{L}_{\text{wp}}
\end{aligned}
\end{equation}

 where $\mathcal{L}_{\text{sp}} $ and $\mathcal{L}_{\text{wp}}  $ denote the masked sentence block prediction loss and the masked word prediction loss respectively.

\subsection{SMITH Model Fine-tuning and Inference}
\label{sec:smith_model_fine_tuning}

After model pre-training, we fine-tune SMITH model on the downstream document matching task with only the binary cross-entropy loss between the estimated matching probability and the ground truth matching label. Note that the word level and sentence block level language modeling masks added during the pre-training stage need to be removed during the fine-tuning stage to avoid losing document content information. After model pre-training and fine-tuning, the trained SMITH model can be used for document representation inference. The document representations inferred by SMITH model offline can be served online with fast similarity search libraries \cite{JDH17,NIPS2017_7157}.

\section{Related Work}
\label{sec:rel}

\textbf{Neural Matching Models.}
A number of neural matching models have been proposed for information retrieval and natural language processing \cite{DBLP:conf/cikm/HuangHGDAH13,DBLP:conf/nips/HuLLC14,DBLP:conf/aaai/PangLGXWC16,Guo:2016:DRM:2983323.2983769,Yang:2016:ARS:2983323.2983818,DBLP:conf/acl/WuWXZL17, Xiong:2017:ENA:3077136.3080809,Mitra:2017:LMU:3038912.3052579,alime-tl}. These models can be classified into the \textit{representation-focused} models and the \textit{interaction-focused} models~\cite{Guo:2016:DRM:2983323.2983769,DBLP:journals/corr/abs-1903-06902}. The \textit{representation-focused} models learn the representations of queries and documents separately, and then they measure the similarity of the representations with functions like cosine, dot, bilinear or tensor layers. On the other hand, the \textit{interaction-focused} models build a query-document word pairwise interaction matrix to capture the exact matching and semantic matching information between query-document pairs. Then a deep neural network which can be a CNN \cite{DBLP:conf/nips/HuLLC14,DBLP:conf/aaai/PangLGXWC16,alime-tl}, term gating network with histogram or value shared weighting mechanism \cite{Guo:2016:DRM:2983323.2983769,Yang:2016:ARS:2983323.2983818} is applied to the query-document interaction matrix to generate the final ranking score. There are also neural matching models which combine the ideas of the \textit{representation-focused} and \textit{interaction-focused} models \cite{Mitra:2017:LMU:3038912.3052579,alime-tl}.
Since it is difficult to serve \textit{interaction-focused} models for online fast inference due to enormous computational costs of the interaction matrices for all query-document pairs, our proposed model belongs to \textit{representation-focused} models, which are also called Siamese models or ``Dual Encoder'' models.

\textbf{Self-Attention Models for Long Text Modeling.}
Self-attention models like Transformer and BERT show promising performance on several tasks in natural language processing and information retrieval. Most of these models are restricted to the representation and matching of short text sequences like sentences and passages. Our work is also built on top of Transformers with a different focus on effective representation learning and matching of long text. Recently there are some related works on adapting Transformers for long text modeling \cite{kitaev2020reformer,qiu2019blockwise,DBLP:journals/corr/abs-1905-06566,DBLP:journals/corr/abs-1901-02860,DBLP:journals/corr/abs-1906-08237,DBLP:journals/corr/abs-1905-07799,roy2020efficient,DBLP:journals/corr/abs-1904-10509,rae2019compressive,ho2019axial}. Zhang et al. \cite{DBLP:journals/corr/abs-1905-06566} proposed the HiBERT model for document summarization and a method to pre-train it using unlabeled data. Our work on the SMITH model is inspired by their research with several differences. First we split the document into sentence blocks with greedy sentence filling for more compact input text structures and less padded words. Second we build a Siamese ``Dual Encoder'' model for document pair similarity modeling. Third we propose a novel pre-training task based on dynamic masked sentence block prediction instead of predicting the masked sentence one word per step as in \cite{DBLP:journals/corr/abs-1905-06566}. We also consider combining representations from different levels of hierarchical Transformers for richer representations.

\textbf{Unsupervised Language Model Pre-training.} The idea of unsupervised learning from plain text for language model pre-training has been explored in several works like Word2Vec\cite{DBLP:conf/nips/MikolovSCCD13}, ELMo\cite{DBLP:journals/corr/abs-1802-05365}, GPT\cite{Radford2018ImprovingLU} and BERT\cite{devlin2018bert}. These models can be pre-trained by predicting a word or a text span using other words within the same sentence. For example,  Word2Vec can be trained by predicting one word with its surrounding words in a fixed text window and BERT pre-trains a language model by predicting masked missing words in a sentence given all the other words. We also study model pre-training techniques on plain text to improve the downstream document matching task. In addition to the masked word prediction task in BERT, we propose the masked sentence block prediction task to learn the relations between different sentence blocks.

redpajama_set_name: RedPajamaArXiv


We've been honest and open up. We glance ahead on your pay a visit to and developing trustworthy and long-term standing relationship. for Big Diameter Astm 252 2800mm Ssaw Steel Pipe Pipes , Big Diameter ASTM 252 2800MM SSAW Steel Pipe Pipes , Big Diameter ASTM 252 40'' Sprial Steel Pipe Pipes , and you will come across our quotation may be very affordable and the top quality of our merchandise is extremely outstanding!
We will usually satisfy our respected shoppers with our superior good quality, fantastic selling price and good company due to we're a lot more expert and far more hard-working and do it in cost-effective way for Big Diameter Astm 252 2800mm Ssaw Steel Pipe Pipes , Big Diameter ASTM 252 2800MM SSAW Steel Pipe Pipes , Big Diameter ASTM 252 40'' Sprial Steel Pipe Pipes , The working experience in the field has helped us forged a strong relations with customers and partners both in domestic and international market. For years our items have been exported to more than 15 countries in the world and have been widely used by customers.


text:
Greeks nervous about coronavirus attack on tourism industry
by Alex Constantine
The Head of the Hellenic Federation of Hoteliers, Mr. Grigoris Tassios said the Greek and international tourism industry are sailing in uncharted waters after the outbreak of COVID19 and the biggest crash test for Greek tourism in 2020 will be the period of the Easter holidays.
Tassios said that there had been no cancellations for Greek tourism destinations to date, aside from those in Santorini and Athens due to the suspension of Air China flights from Beijing to the Greek capital.
"Something like this is not apparent at the moment, but if it does happen during the current period there are margins for recovery and correction. If, however, (the spread of the virus) peaks in April, then there will be a serious problem," he noted.
For the time being, the prevailing message in the Greek tourist sector is one of calmness and cool headedness, at the same time the tourism ministry is on alert and cooperating closely with health services and organisations for the coordination of their action.
In the next few months, Greece is expected to welcome over 30 million visitors and this demands calmness, coordination and cooperation from all, it noted.
"SETE is in constant communication with its members, the authorities and state agencies in order to ensure an assessment of the situation, prompt information and compliance with the prevention measures that need to be taken," the announcement said.
Tags: Air China, Athens, Beijing, coronavirus, COVID19, Easter, greece, Greek news, Grigoris Tassios, Hellenic Federation of Hoteliers, Santorini, Tourism, travel
'Marti': Greece's customary red and white bracelet to welcome Spring
Greek Lessons OnLine: Your Beautiful Online Family
Alex Constantine March 2, 2020
1,400-year-old burial stone with Greek inscription discovered in Israel
Gct January 7, 2021
Greek passport can access 184 countries visa-free in 2021
GCT B January 7, 2021
Greek PM's daughter helps prepare meals for the homeless in Attica
Aegean Airlines resume flights to Yerevan with discount sale
Due to the Turkish-sponsored invasion of Artsakh by the Azerbaijani military and Syrian mercenaries, direct…
President Sakellaropoulou: "Respect for rules is the essence of democracy"
Greek President Katerina Sakellaropoulou spoke about a dark moment in American history after yesterday's events…
Mayor of Kalymnos explains how the incident with the Turkish Coast Guard in Imia took place
The incident between Greek fishermen, the Turkish coastguard and the Greek Coast Guard was explained…
Increased security around the American embassy in Athens
After the chaos in Washington, authorities considered it appropriate to increase security measures around the…
Patriarch of Moscow: Conversion of Hagia Sophia into a mosque is divine punishment against
A provocative statement by the Patriarch of Moscow Kirill during a TV interview he gave…

redpajama_set_name: RedPajamaCommonCrawl


text:
South LA #StepsUp TO
THE PEOPLE'S STORIES
SHARE YOUR COVID-19 STORY
POLITICS AND PARTNERSHIPS
Community Coalition (CoCo) was built to help transform the social and economic conditions of South LA residents—from the crack cocaine epidemic of the 1980s to the challenges impacting South Central amidst the COVID-19 global crisis. CoCo stepped up then and we are stepping up now to build power for our community, change policy and promote homegrown leadership development.
In alignment with government directives to implement social distancing, our Community Coalition team will maintain its office closure and virtual operations schedule. Although our office is physically closed, our team is working remotely and implementing programs through virtual mediums. We remain relentlessly committed to our members and are communicating daily with our base via wellness and needs check-ins to offer mutual aid, stay socially connected, strengthen political partnerships and keep members aware of CoCo's current campaigns.
We fought for a People's Budget!
Our members voiced their concerns and urged the Los Angeles City Council to delay passing a city budget that included a $123 million increase to the Los Angeles Police Department and disrupts sorely needed city services that are vital to the community.
Supplies and Resources
Since March 16, CoCo has been delivering water, sanitizing supplies, paper towels and toilet paper to our members. We've talked to more than 1,000 members and distributed supplies to more than 50 families.
Community Coalition is honored to join forces with the Brotherhood Crusade and InnerCity Struggle to fight the digital divide and address the learning needs of our students. The three organizations joined forces to raise $400,000 to support 5,000 high need South and East Los Angeles students with technology to address deep educational inequities in our community. Click here for more information.
Our small businesses are the backbone of South LA. They're the ones that need our financial support right now, not the big corporate chains. Where you can, consider supporting small businesses during this time. Here are a few ways that you can support some of the ones we collaborate with in our South Central by South Central (SCxSC) "Development By Us & For Us" campaign.
South LA Cafe
Make a One-Time PayPal Donation to South LA Cafe
If you're an essential worker who's out and about, consider placing a take-out order. To do so, download the Joe Coffee App by texting "coffee" to 474747. When you check out, make sure to click curbside delivery. Joe and Celia will bring it right to your car!
Village Market Place
Online Ordering with Delivery
Mercado La Paloma
Family Meals from Chichen Itza Restaurant
The Tree South LA
Our friends there are offering Instagram Live yoga sessions. Check out their Instagram for updates on when classes are coming up. The suggested donation is $10 to their Venmo (@TheTreeSouthLA) or PayPal (@GTYM)
Mind, Body, Soul Activities & Resources
We've compiled a list of Mind, Body, Soul resources for you to tap into and use as needed. The isolation and fallout from COVID-19 can lead to high levels of stress and anxiety. Nurturing our mental, physical, and spiritual wellbeing is an important way to stay healthy.
CoCo's Mind, Body, Soul Resources
South Central Youth Empowered thru Action (SCYEA) wants to hear about your experiences during the LAUSD school closure. To uplift student voices and experiences in South LA schools, SCYEA will work to share as many different student perspectives as possible. Youth leaders will use that data to inform courses of action that will improve the student experience. You can join this movement and advocate on behalf of yourself and your peers for better access to resources from LAUSD!
PLEASE SHARE THIS SURVEY WITH OTHER STUDENTS
Click here to take the Survey
As we prepare for November's election, we've seen Black and Brown voting rights come under attack cross the nation. With so much on the line locally and nationally for communities of color, we are committed to battling voter suppression in all its forms. From the lack of voting center locations in Los Angeles to fighting the Republican National Committee when it tried to deny citizens the right to vote-by-mail, Community Coalition remains vigilant and ready to take action.
SENIOR MEAL PROGRAM
Councilmember Marqueece Harris-Dawson launched an emergency meal program to support Council District 8 residents over the age of 65 during the COVID-19 safe at home order. We know our seniors are most vulnerable and we are teaming up with local, family-owned businesses to deliver FREE meals to seniors in our District.
You must be a Council District 8 resident to participate in this program. If you are interested in receiving free meals through the Emergency Senior Meals Program, call 213.485.7616 to join the waitlist. You will receive a call to let you know when you can begin receiving meals.
If you are a business interested in participating, call 213.485.7616.
Accessing the Resources in the Stimulus Package
How can you benefit from the stimulus package passed by Congress? This Q&A from Congressmember Karen Bass will walk through who qualifies for the cash payment, unemployment insurance, emergency leave, and more.
Q&A on the Stimulus Package: CARES Act
Covid-19 Resources - Congressmember Karen Bass
Rent Freeze for Los Angeles
Community Coalition has talked to 1,000 of our members, and many of them are concerned about the possibility of being homeless as the impact of the COVID-19 pandemic continues to be felt.
The city struggles to house more than 44,000 each night who sleep on the street.
Thirteen thousand seven hundred nineteen are Black, and 12,442 are Brown community members.
In SPA 6, which encompasses South LA, there were 9,629 residents reported as homeless last year.
In L.A. County, 34% of the homeless residents in 2019 were Brown.
Black people make up 8 percent of the County's population, but 31% percent of the homeless population.
Although Governor Newsom recently announced an eviction moratorium for the state of California, the executive order is actually only a delay. Property owners can still initiate evictions and tenants can still be put out of their homes after May 31 for failure to pay during the months of April and May. On March 30, 11th District Los Angeles City Councilmember Mike Bonin proposed a broader ban on evictions which fell one vote short of adoption. We need city officials not only to implement an eviction moratorium, but also to take it one step further and introduce rent forgiveness.
Community Coalition applauds Supervisor Mark Ridley-Thomas for introducing the most recent amendment to deepen tenant protections by implementing a rent freeze across the County. The Board of Supervisors' unanimous approval further demonstrates that they are doing everything in their power to keep residents in their homes during the COVID-19 crisis. We know it will take members of the community, especially low-income renters, months to recover economically from the COVID crisis. And while the rent freeze goes a long way, we want to make sure those who are most impacted will not be burdened by the added stress of unrealistic repayment obligations.
I live with my mom in a one bedroom apartment. It's $900 in rent per month. I make just enough each month to be able to pay rent, utilities and the phone bill while also going to school and taking care of my mom. I've been putting money aside since last year just in case anything were to happen so that we have a cushion, but I'm worried about the quarantine forcing me to burn through my savings.
– Joseph Hernandez, 19
We need you to get involved.
Let your Councilmember know your concerns regarding COVID-19's threat of leaving members of our community homeless/houseless by recording a video and uploading it to your Councilmember's media pages.
Example: "I see community members who are houseless daily; I am afraid I will be homeless too if a rent forgiveness program isn't put in place in response to COVID-19."
01 Record your COVID-19 story and tag your CITY COUNCILMEMBER and @LACityCouncil
02 Find your Councilmember's Social Media Accounts here
CoCo's Current Campaigns
The Census: Strengthening Our Social Safety Net
Historically, communities of color have been undercounted. Step up to make sure our kids and communities are counted to receive the benefits of the census. Covid-19 has changed how we make sure our people are counted in the census so our communities get the funding and resources we deserve. The census is an opportunity to bolster our public health infrastructure, the lack of which is being exposed right now.
A once every 10 years, 10 minute, 10 question survey, the census has major implications in the amount of political representation and funding a community receives for medical clinics, schools, job programs, and public transportation. While we have the time, let's take the time to fill out the census.
Take the 2020 Census Survey
CoCo's Census Tutorial
The activism of Community Coalition members and community residents has been a steadfast catalyst for aid in South LA for decades. Anything you're able to give to CoCo right now would be appreciated in our continued efforts to improve the material conditions in South LA as we respond to the community's needs during and after the Corona crisis.


Seven Signs is a comprehensive evangelism experience designed to bring people to a decision for Jesus and prepare them for a life-long walk with Him. This five-DVD set includes eight 60-minute presentations by Pastor Jose Rojas. Also included in the DVD set are promotional materials your church can use to prepare for the Seven Signs series.


text:
https://www.naval-technology.com/news/fincantieri-to-build-two-new-generation-submarines-for-italian-navy/ 20% Daily 2021-03-01 13:55
https://www.naval-technology.com/news/uk-mod-awards-funding-to-rolls-royce-for-naval-autonomy-technology/ 20% Daily 2021-02-26 12:51
https://www.naval-technology.com/news/us-destroyer-uss-curtis-wilbur-conducts-routine-taiwan-strait-transit/ 20% Daily 2021-02-26 12:50
https://www.naval-technology.com/news/frcsw-to-conduct-e-2d-pmi-review-under-nss-in-partnership-with-bcg/ 20% Daily 2021-08-13 10:21
https://www.naval-technology.com/news/us-dod-serdp-awards-funding-to-nps-for-new-research-project/ 20% Daily 2021-02-25 12:11
https://www.naval-technology.com/news/secon-to-build-missile-cum-ammunition-barge-for-indian-navy/ 20% Daily 2021-02-25 12:35
https://www.naval-technology.com/news/russian-conventional-platforms-are-significant-drivers-of-defence-expenditure-in-western-europe-poll/ 20% Daily 2021-02-25 10:07
https://www.naval-technology.com/news/nato-begins-dynamic-manta-2021-anti-submarine-warfare-exercise/ 20% Daily 2021-02-24 16:48
https://www.naval-technology.com/news/british-royal-navy-contracts-saab-for-digital-tower-solution/ 20% Daily 2021-02-24 16:48
https://www.naval-technology.com/news/saic-to-deliver-fuel-tanks-for-us-navys-mk48-heavyweight-torpedo/ 20% Daily 2021-02-24 16:48
https://www.naval-technology.com/news/cnrs-creates-lab-in-australia-to-focus-on-human-robotic-interfaces/ 20% Daily 2021-02-23 16:56
https://www.naval-technology.com/comment/naval-tech-trends-submarine-2/ 20% Daily 2021-03-01 09:05
https://www.naval-technology.com/news/indias-drdo-conducts-two-launches-of-vl-srsam-missile/ 20% Daily 2021-02-23 13:38
https://www.naval-technology.com/news/hms-spey-to-begin-next-testing-phase-in-solent-under-white-ensign/ 20% Daily 2021-02-23 13:38
https://www.naval-technology.com/news/france-third-generation-ssbn-submarine-programme/ 20% Daily 2021-02-22 17:15
https://www.naval-technology.com/news/hii-uss-john-c-stennis-aircraft-carrier-rcoh/ 20% Daily 2021-02-22 15:02
https://www.naval-technology.com/news/first-f-35b-aircraft-laser-peening/ 20% Daily 2021-02-22 12:44
https://www.naval-technology.com/features/first-sea-lord-talks-future-operations-surface-warships-2021/ 20% Daily 2021-03-05 09:16
https://www.naval-technology.com/news/uss-porter-romanian-navy-airforce-exercises/ 20% Daily 2021-02-19 17:22
https://www.naval-technology.com/news/advanced-materials-technology-use-in-naval-shipbuilding-should-focus-on-stealth-poll/ 20% Daily 2021-02-19 13:11
https://www.naval-technology.com/news/usmc-receives-amphibious-combat-vehicle-command-variant/ 20% Daily 2021-02-22 05:49
https://www.naval-technology.com/news/ultra-hull-mounted-sonar-canadas-csc-project/ 20% Daily 2021-02-19 12:36
https://www.naval-technology.com/sponsored/information-advantage-smart-technology-empowering-smart-people/ 20% Daily 2022-05-10 16:02
https://www.naval-technology.com/news/australia-rigid-hull-inflatable-boat-bahrain-defence-force/ 20% Daily 2021-02-18 17:31
https://www.naval-technology.com/news/astronics-to-deliver-power-distribution-units-for-boeing-mq-25-uas/ 20% Daily 2021-02-19 16:12
https://www.naval-technology.com/news/us-navy-destroyer-fonop-south-china-sea/ 20% Daily 2021-02-18 12:48
https://www.naval-technology.com/sponsored/weapon-alignment-technology-accuracy-reliability/ 20% Daily 2022-11-09 10:26
https://www.naval-technology.com/news/eca-groups-umisas-sonar-successfully-completes-sea-trials/ 20% Daily 2021-02-17 17:17
https://www.naval-technology.com/news/ixblue-to-deliver-navigation-systems-for-spanish-f110-class-frigates/ 20% Daily 2021-02-17 12:48
https://www.naval-technology.com/news/us-approves-sale-of-168-ram-block-2-tactical-missiles-to-egypt/ 20% Daily 2021-02-17 12:48
https://www.naval-technology.com/news/titagarh-wins-indian-navy-contract-for-five-diving-support-craft/ 20% Daily 2021-02-16 17:50
https://www.naval-technology.com/news/hensoldt-upgrades-trs-3d-radars-of-germany-navys-two-k130-corvettes/ 20% Daily 2021-02-16 13:19
https://www.naval-technology.com/news/vectrus-wins-contract-for-naval-base-coronado-5g-smart-warehouse/ 20% Daily 2021-02-16 13:19
https://www.naval-technology.com/news/fincantieri-qatar-damsah/ 20% Daily 2021-02-15 16:07
https://www.naval-technology.com/news/indian-mod-lrsam-missiles/ 20% Daily 2021-02-15 13:43
https://www.naval-technology.com/news/aman-2021-arabian-sea/ 20% Daily 2021-02-15 13:43
https://www.naval-technology.com/news/airbus-subsidiary-to-supply-aliaca-fixed-wing-drone-systems-to-french-navy/ 20% Daily 2021-02-15 13:43
https://www.naval-technology.com/news/bae-systems-wins-usmc-contract-option-for-more-acvs/ 20% Daily 2021-02-15 13:43
https://www.naval-technology.com/news/navantia-and-naviris-partner-for-european-patrol-corvette-project/ 20% Daily 2021-02-12 11:42
https://www.naval-technology.com/news/nswc-phd-and-ucsb-partner-for-automated-corrosion-testing-system/ 20% Daily 2021-02-11 18:02
https://www.naval-technology.com/news/bae-to-deliver-naval-guns-for-belgian-dutch-mcmv-programme/ 20% Daily 2021-02-11 12:23
https://www.naval-technology.com/news/sikorsky-wins-contract-for-final-lot-of-vh-92a-presidential-helicopters/ 20% Daily 2021-02-12 14:06
https://www.naval-technology.com/features/fire-scout-asw/ 20% Daily 2021-02-11 11:55
https://www.naval-technology.com/news/cae-pinnacle-solutions-jv-xebec-to-deliver-first-lcs-bptts-to-us-navy/ 20% Daily 2021-02-10 17:44
https://www.naval-technology.com/comment/submarine-uss-asheville-naval-tech/ 20% Daily 2021-02-10 13:27
https://www.naval-technology.com/news/us-carrier-strike-groups-exercises-in-south-china-sea/ 20% Daily 2021-02-10 14:53
https://www.naval-technology.com/news/hms-lancaster-tests-heavy-machine-guns/ 20% Daily 2021-02-11 10:05
https://www.naval-technology.com/sponsored/qa-maritime-vessels-marine-alutech/ 20% Daily 2022-05-10 16:13
https://www.naval-technology.com/news/indias-mod-and-bharat-electronics-sign-sdr-tac-procurement-contract/ 20% Daily 2021-02-09 15:56
https://www.naval-technology.com/news/fincantieri-selects-l3-harris-to-support-us-navys-ffg-62-frigate/ 20% Daily 2021-02-09 13:04
https://www.naval-technology.com/news/hms-queen-elizabeth-undergoes-virtual-warrior-assessment/ 20% Daily 2021-02-09 14:28
https://www.naval-technology.com/news/us-darpa-contractors-unmanned-underwater-vehicle-project/ 20% Daily 2021-02-08 17:24
https://www.naval-technology.com/news/british-royal-navy-hms-talent-tests-spearfish-torpedo/ 20% Daily 2021-02-08 12:29
https://www.naval-technology.com/news/chile-requests-sale-of-sm-2-block-iiia-missiles-from-us/ 20% Daily 2021-02-08 12:57
https://www.naval-technology.com/news/us-uk-and-saudi-naval-forces-conclude-exercise-nautical-defender-21/ 20% Daily 2021-02-05 17:26
https://www.naval-technology.com/news/indian-mod-opens-upgraded-naval-environmental-test-facility/ 20% Daily 2021-02-05 12:48
https://www.naval-technology.com/news/leonardo-to-provide-new-hf-radio-system-for-us-navys-e-2d-fleet/ 20% Daily 2021-02-05 12:28
https://www.naval-technology.com/news/ultra-to-provide-tlfas-variable-depth-sonar-system-for-canadas-csc/ 20% Daily 2021-02-04 17:16
https://www.naval-technology.com/news/damen-and-den-breejen-sign-rnln-new-css-preservation-contract/ 20% Daily 2021-02-04 12:35
https://www.naval-technology.com/news/norfolk-naval-shipyard-staff-creates-air-filtration-units/ 20% Daily 2021-02-04 12:31
https://www.naval-technology.com/news/indian-navys-ilyushin-38sd-aircraft-fires-missiles-during-tropex-21/ 20% Daily 2021-02-03 17:11
https://www.naval-technology.com/news/northrop-and-ultra-demonstrate-unmanned-aircraft-system-mission/ 20% Daily 2021-02-03 13:07
https://www.naval-technology.com/news/uk-royal-marines-deploy-to-arctic-for-intensive-training/ 20% Daily 2021-02-03 13:07
https://www.naval-technology.com/news/us-navy-flight-tests-tcts-ii-air-combat-training-system-on-f-a-18f/ 20% Daily 2021-02-03 13:07
https://www.naval-technology.com/features/safeguarding-shipping-in-the-middle-east-imsc/ 20% Daily 2021-02-02 16:46
https://www.naval-technology.com/news/bae-systems-secures-us-navy-contract-to-sustain-atc-systems/ 20% Daily 2021-02-02 17:03
https://www.naval-technology.com/news/hii-wins-aircraft-carrier-support-contract-from-us-navy/ 20% Daily 2021-02-02 17:03
https://www.naval-technology.com/news/iai-to-sell-harop-system-naval-version-to-undisclosed-asian-country/ 20% Daily 2021-02-02 17:03
https://www.naval-technology.com/news/uss-toledo-submarine-undergo-engineered-overhaul/ 20% Daily 2021-02-01 15:10
https://www.naval-technology.com/news/philippine-navy-accepts-second-rizal-class-frigate/ 20% Daily 2021-02-01 15:10