text:
\section{Introduction}

There has been a wealth of research over the past several years on self-supervised pre-training for natural language processing tasks~\cite{Devlin2019BERT,liu2019roberta,Conneau2020xlmr,jiao2020exploiting}, which aims at transferring the knowledge of large-scale unlabeled data to downstream tasks with labeled data.
Despite its success in other understanding and generation tasks, self-supervised pretraining is not a common practice in machine translation (MT).
One possible reason is the architecture discrepancy between pretraining model (e.g., Transformer {\bf encoder}) and NMT models (e.g., Transformer {\bf encoder-decoder}).

To remedy the architecture gap, several researchers propose sequence-to-sequence (Seq2Seq) pretraining models for machine translation, e.g., MASS~\cite{song2019mass} and BART~\cite{zhu2019incorporating,lewis2020bart}.
Recently,~\newcite{Liu2020mbart} extend BART by training on large-scale multilingual language data (i.e., mBART), leading to significant improvement on translation performance across various language pairs.
While previous pretraining approaches for NMT generally focus only on Transformer encoder~\cite{Lample2019xlm}, mBART pretrains a complete autoregressive Seq2Seq model by recovering the input sentences that are noised by masking phrases.
One research question naturally arises: {\em how much does the jointly pretrained decoder matter?}

In this work, we present a substantial step in better understanding the SOTA Seq2Seq pretraining model. We take a fine-grained look at the impact of the jointly pretrained decoder by carefully designing experiments, which are conducted on several WMT and IWSLT benchmarks across language pairs and data scales using the released mBART-25 model~\cite{Liu2020mbart}.
By carefully examining the translation outputs, we find that ($\S~\ref{sec:translation-performance}$):
\begin{itemize}[leftmargin=10pt]
    \item Jointly pretraining decoder produces more diverse translations with different word orders, which calls for multiple references to accurately evaluate its effectiveness on large-scale data.
    \item Jointly pretraining decoder consistently reduces adequacy-related translation errors over pretraining encoder only.
\end{itemize}

Although jointly pretraining decoder consistently improves translation performance, we also identify several side effects due to the discrepancies between pretraining and finetuning (\S\ref{sec:pretrain-finetune-gaps}):
\begin{itemize}[leftmargin=10pt]
    \item {\bf domain discrepancy}: Seq2Seq pretraining model is generally trained on general domain data while the downstream translation models are trained on specific domains (e.g., news). The domain discrepancy requires more efforts for the finetuned model to adapt the knowledge in pretrained models to the target in-domain.
    \item {\bf objective discrepancy}: NMT training learns to translate a sentence from one language to another, while Seq2Seq pretraining learns to reconstruct the input sentence. The objective discrepancy induces the over-estimation issue and tends to generate more hallucinations with noisy input. The over-estimation problem along with more copying translations induced by Seq2Seq pretraining~\cite{Liu:2021:ACL} make it suffer from more serious beam search degradation problem.
\end{itemize}

To remedy the above discrepancies, we propose simple and effective strategies, named  in-domain pretraining and input adaptation in finetuning (\S\ref{sec:approach}). In in-domain pretraining, we propose to reduce the domain shift by continuing the pretraining of mBART on in-domain monolingual data, which is more similar in data distribution with the downstream translation tasks.
For input adaptation, we add noises to the source sentence of bilingual data, and combine the noisy data with the clean bilingual data for finetuning. We expect the perturbed inputs to better transfer the knowledge from pretrained model to the finetuned model.
Experimental results on the benchmark datasets show that in-domain pretraining improves the translation performance significantly and input adaptation enhances the robustness of NMT models. Combining the two approaches gives us the final solution to a well-performing NMT system. Extensive analyses show that our approach can narrow the domain discrepancy, particularly improving the translation of low-frequency words. Besides, our approach can alleviate the over-estimation issue and mitigate the beam search degradation problem of NMT models.

\begin{table*}[t!]
\centering
\setlength{\tabcolsep}{5pt}
\begin{tabular}{ccc cc cc cc cc cc cc}
\toprule
\multicolumn{3}{c}{\bf Pretraining}
& \multicolumn{2}{c}{\bf W19 En-De}
& \multicolumn{2}{c}{\small \bf W19 En-De (S)}
& \multicolumn{2}{c}{\bf W16 En-Ro}
& \multicolumn{2}{c}{\bf I17 En-Fr} \\
\cmidrule(lr){1-3} \cmidrule(lr){4-5}  \cmidrule(lr){6-11}
\em Model   &   \em {Enc} & \em {Dec}
&  \em $\Rightarrow$  &   \em $\Leftarrow$ &  \em $\Rightarrow$  &   \em $\Leftarrow$ &  \em $\Rightarrow$  &   \em $\Leftarrow$ &  \em $\Rightarrow$  &   \em $\Leftarrow$  \\
\midrule
\multicolumn{3}{c}{\bf no pretrain}  &  39.6    &   41.0   & 29.7 & 30.1   & 34.5 &   34.3   & 37.3 & 38.0   \\
\midrule
\multirow{3}{*}{\bf mBART}  &   \texttimes  &   \texttimes     & 39.4  & 40.1  & 26.7 &  27.1  &   30.0 & 29.6 & 35.3 & 35.1 \\
   & \checkmark    &   \texttimes  & \cellcolor{red!11.2} 40.8 & \cellcolor{red!8} 41.1  &  \cellcolor{red!64} 31.7 & \cellcolor{red!51.2} 33.5 & \cellcolor{red!40} {35.0} & \cellcolor{red!48} {35.6} &
  \cellcolor{red!62} 38.4 & \cellcolor{red!66}38.4 \\
  &   \checkmark    &   \checkmark   & \cellcolor{red!11.2} 40.8 & \cellcolor{red!10.4} 41.4 & {\cellcolor{red!}} 35.3 & {\cellcolor{red!84.6}} 35.7 &  \cellcolor{red!60} 37.1 & \cellcolor{red!62.4} 37.4 &
\cellcolor{red!80}39.2 & \cellcolor{red!99}40.2 \\
\bottomrule
\end{tabular}
\caption{BLEU scores on MT benchmarks. ``Enc:\texttimes, Dec:\texttimes'' represents that we use only the pre-trained embeddings for fair comparisons, and we highlight performance improvement over this setting in {\color{zred}red} color.}
\label{tab:benchmarks}
\end{table*}

\section{Understanding Seq2Seq Pretraining}

In this section, we conduct experiments and analyses to gain a better understanding of current Seq2Seq pretraining for NMT. We first present the translation performance of the pretrained components (\S\ref{sec:translation-performance}), and then show the discrepancy between pretraining and finetuning (\S\ref{sec:pretrain-finetune-gaps}).

\subsection{Experimental Setup}

\paragraph{Data.}
We conduct experiments on several benchmarks across language pairs, including high-resource WMT19 English-German (W19 En-De, 36.8M instances), and low-resource WMT16 English-Romanian (W16 En-Ro, 610K instances) and IWSLT17 English-French (I17 En-Fr, 250K instances).
To eliminate the effect of different languages, we also sample a subset from WMT19~En-De (i.e., W19~En-De~(S), 610K instances) to construct a low-resource setting for ablation studies.

For the proposed {\em in-domain pretraining}, we collect the NewsCrawl monolingual data as the in-domain data for WMT tasks (i.e., 200M English, 200M German, and 60M Romanian), and the TED monolingual data for IWSLT tasks (i.e., 1M English and 0.9M French). Since the monolingual data from TED is rare, we expand it with pseudo in-domain data, OpenSubtitle~\cite{tiedemann2016finding}, which also provides spoken languages as TED. Specifically, we use the latest 200M English subtitles and all the available French subtitles (i.e., 100M).
We follow~\newcite{Liu2020mbart} to use their released sentence-piece model~\cite{kudo-richardson-2018-sentencepiece} with 250K subwords to tokenize both bilingual and monolingual data.
We evaluate the translation performance using the SacreBLEU~\cite{post-2018-call}.

\paragraph{Models.}
As for the pretrained models, we adopt the officially released mBART25 model~\cite{Liu2020mbart}\footnote{\url{https://github.com/pytorch/fairseq/tree/main/examples/mbart}}, which is trained on the large-scale CommonCrawl~(CC) monolingual data in 25 languages.
As a result, the vocabulary is very large in mBART25, including 250K words.
mBART uses a larger Transformer model which extends both the encoder and decoder of Transformer-Big to 12 layers. We use the parameters of either encoder or encoder-decoder from the pretrained mBART25 for finetuning. Then, in the following section, we use pretrained encoder, and pretrained encoder-decoder for short.
We follow the officially recommended finetuning setting with dropout of $0.3$, label smoothing of $0.2$, and warm-up of $2500$ steps. We finetune on the high-resource task for 100K steps and the low-resource tasks for 40K steps, respectively.

We also list the results of vanilla Transformer without pretraining as baseline.
The vocabulary is built on the bilingual data, hence is much smaller (e.g., En-De 44K) than mBART25.
Specifically, for high-resource tasks we train 6L-6L Transformer-Big with 460K tokens per batch for 30K steps, and for low-resource tasks we train 6L-6L Transformer-Base with 16K tokens per batch for 50K steps.

\subsection{Impact of Jointly Pretrained Decoder}
\label{sec:translation-performance}

The main difference of Seq2Seq pretraining models (e.g., mBART) from previous pretraining models (e.g., BERT and XLM-R) lies in whether to train the decoder together.
In this section, we investigate the impact of the jointly pretrained decoder in terms of BLEU scores, and provide some insights on where the jointly pretrained decoder improves performance.

\paragraph{Translation Performance.}
Table~\ref{tab:benchmarks} lists the BLEU scores of pretraining different components of NMT models, where we also include the results of NMT models trained on the datasets from scratch (``no pretrain'').
For fair comparisons, we use the same vocabulary size for all variants of pretraining NMT components.
We use the pretrained word embedding for the model variant with randomly initialized encoder-decoder (``Enc:\texttimes, Dec:\texttimes''), which makes it possible to train 12L-12L NMT models on the small-scale datasets. Accordingly, the results of (``Enc:\texttimes, Dec:\texttimes'') is worse than the ``no pretrain'' model due to the larger vocabulary (e.g., 250K vs. 44K) that makes the model training more difficult.

Pretraining encoder only (``Enc:\checkmark, Dec:\texttimes'') significantly improves translation performance, which is consistent with the findings in previous studies~\cite{zhu2019incorporating,weng2020acquiring}.
{We also conduct experiments with the pretrained encoder XLM-R~\cite{Conneau2020xlmr}, which achieves comparable performance as the mBART encoder (see Appendix~\ref{sec:XLMR}). For fair comparisons, we only use the mBART encoder in the following sections.}
Encouragingly, jointly pretraining decoder can further improve translation performance, although the improvement is not significant on the large-scale WMT19 En-De data. These results seem to provide empirical support for the common cognition -- pretraining is less effective on large-scale data. However, we have some interesting findings of the generated outputs, which may draw different conclusions.
To eliminate the effect of language and data bias, we use the full set and sampled subset of WMT19 De$\Rightarrow$En data as representative large-scale and small-scale data scenarios.

\begin{table}[t]
\centering
\small
\setlength{\tabcolsep}{5pt}
\begin{tabular}{c m{6cm}}
\toprule
\em Src  & Sie bezichtigt die Erwachsenen Kinderhandel zu betreiben.  \\
\em Ref   &  She accuses the adults of child trafficking. \\
\midrule
\multicolumn{2}{c}{\bf Large-Scale Data}\\
no pre. &  {\color{red}It} accuses {\color{red}(the)} adults of children trafficking. \\
(\texttimes, \texttimes)    &  {\color{red}It} accuses {\color{red}(the)} adults of children trafficking.  \\
(\checkmark, \texttimes)    &  She accuses the adults of children trafficking. \\
(\checkmark, \checkmark)    & She accuses the adults of {\color{blue}trafficking in children}.
  \\
\midrule
\multicolumn{2}{c}{\bf Small-Scale Data}\\
no pre. &  {\color{red}It} accuses the adults {\color{red}to trade} children. \\
(\texttimes, \texttimes) &  {\color{red}It requires adult trafficking on children. }\\
(\checkmark, \texttimes) & {\color{red}It} accuses {\color{red}(the)} adults of children trafficking. \\
(\checkmark, \checkmark)    &  She accuses the adults of {\color{blue}trafficking in children}. \\
\bottomrule
\end{tabular}
\caption{Translation examples on WMT19 De$\Rightarrow$En test set. The translation errors are highlighted in {\color{red}red} and changes of word order are highlighted in {\color{blue}blue}.}
\label{tab:case-study}
\end{table}

Table~\ref{tab:case-study} shows some translation examples. Firstly, jointly pretraining decoder can produce good translations that are different in the word order from the ground-truth reference (e.g., ``trafficking in children'' vs. ``child trafficking"), thus are assigned low BLEU scores.
This may explain why jointly pretraining decoder only marginally improves performance on large-scale data.
Secondly, jointly pretraining decoder can reduce translation errors, especially on small-scale data (e.g., correct the mistaken translation of ``{\em It}'' to ``{\em She}'').
We empirically validate the above two findings in the following experiments.

\begin{table}[t]
\centering
\fontsize{10}{12}\selectfont
\begin{tabular}{c lr lr}
\toprule
\multirow{2}{*}{\bf Pretrain}   &   \multicolumn{2}{c}{\bf Single}   &   \multicolumn{2}{c}{\bf Multiple}\\
\cmidrule(lr){2-3}  \cmidrule(lr){4-5}
  &   BLEU  &  $\triangle$ &   BLEU    &   $\triangle$ \\
\midrule
\multicolumn{5}{l}{\bf Large-Scale Data}\\
no pretrain               &   39.5    & -   &   77.1    &   -\\
\hdashline
(\texttimes, \texttimes)  &   38.6    & -0.9    &   75.7    &   -1.4\\
(\checkmark, \texttimes)  &   39.5    & +0.0    &   77.8    &   +0.7\\
(\checkmark, \checkmark)  &   39.9    & +0.4    &   79.1$^{\Uparrow}$    &   \bf +2.0\\
\midrule
\multicolumn{5}{l}{\bf Small-Scale Data}\\
no pretrain   &  27.0    &  -   &   53.1    &   -\\
\hdashline
(\texttimes, \texttimes)  &  27.0   &   +0.0 &  52.3    &   -0.8\\
(\checkmark, \texttimes)  &  32.3   &   +5.3 &  63.4    &   +10.3\\
(\checkmark, \checkmark)  &  35.3$^{\Uparrow}$    &   +8.3    &   69.1$^{\Uparrow}$    &   \bf +16.0\\
\bottomrule
\end{tabular}
\caption{BLEU scores on En$\Rightarrow$De testset with single and multiple references. ``$\Uparrow$'' denotes significantly better (with $p < 0.01$) than No mBART pretraining.}
\label{tab:multi-reference}
\end{table}

\paragraph{Impact on Translation Diversity.}
We follow~\newcite{Du:2021:ICML} to better evaluate the translation quality for different word orders using multiple references.
We use the test set released by~\citet{ott2018analyzing}, which consists of 10 human translations for 500 sentences taken from the WMT14 En$\Rightarrow$De test set.
As shown in Table~\ref{tab:multi-reference}, the pretrained decoder achieves more significant improvement in all cases when measured by multiple references. These results provide empirical support for our claim that jointly pretraining decoder produces more diverse translations with different word orders, which can be better measured by multiple references.
These results may renew our cognition of pretraining, that is, {\em they are also effective on large-scale data when evaluated more accurately}.

\begin{table}[t!]
\fontsize{10}{12}\selectfont
\centering
\begin{tabular}{cc rr rr rr}
\toprule
\multicolumn{2}{c}{\bf Pretrain}   &   \multicolumn{3}{c}{\bf Large}   &   \multicolumn{3}{c}{\bf Small}\\
\cmidrule(lr){1-2}  \cmidrule(lr){3-5}  \cmidrule(lr){6-8}
\em {Enc} & \em {Dec} & \em Ut   &   \em Mt &  \em Ot & \em Ut   &   \em Mt &  \em Ot\\
\cmidrule(lr){1-2}  \cmidrule(lr){3-8}
\texttimes  &   \texttimes & 4 & 9 & 0 & 25 & 45 & 0 \\
\checkmark  &   \texttimes & 3 & 3 & 0 & 5 & 21 & 5 \\
\checkmark  &   \checkmark & 2 & 0 & 0 & 3 & 15 & 0 \\
\bottomrule
\end{tabular}
\caption{Human evaluation of mBART pretrained NMT models in terms of under-translation (Ut), mis-translation (Mt), and over-translation (Ot) errors.}
\label{tab:mbart-adequacy}
\end{table}

\paragraph{Impact on Adequacy.}
We conduct a human evaluation to provide a more intuitive understanding of how jointly pre-training decoder improves translation quality. Specifically, we ask two annotators to annotate under-translation, mis-translation and over-translation on 100 sentences randomly sampled from WMT19 De$\Rightarrow$En test set. As listed in Table~\ref{tab:mbart-adequacy}, inheriting the pretrained decoder reduces more translation errors on small data than on large data, which is consistent with the results of BLEU score in Table~\ref{tab:benchmarks}.
Interestingly, inheriting only the pretrained encoder introduces more over-translation errors on small data, which can be solved by combining the pretrained decoder. One possible reason is that inheriting only the pretrained encoder excessively enlarges the impact of source context.\footnote{\newcite{Tu:2017:TACL} showed that more impact of source context leads to over-translation errors.}
This problem does not happen on large data, since the large amount of in-domain data can balance the relation between encoder and decoder to accomplish the translation task well.

\subsection{Pretraining-and-Finetuning Discrepancy}
\label{sec:pretrain-finetune-gaps}

Although Seq2Seq pretraining consistently improves translation performance across data scales, we find several side effects of Seq2Seq pretraining due to the discrepancy between pretraining and finetuning.
In this section, we present two important discrepancies: {\em domain discrepancy} and {\em objective discrepancy}. Unless otherwise stated, we report results on WMT19 En-De test set using small data.

\subsubsection{Domain Discrepancy}
\label{sec:domain-discrepancy}

Seq2Seq pretraining model is generally trained on {\bf general domain} data while the downstream translation models are trained on {\bf specific domains} (e.g., news). Such a domain discrepancy requires more efforts for the finetuned models to adapt the knowledge in pretrained models to the target in-domain. We empirically show the domain discrepancy in terms of lexical distribution and domain classifier.

\begin{figure}[t!]
    \centering
    \includegraphics[height=0.3\textwidth]{Figures/lexicon.pdf}
    \caption{Word distributions of English corpora from general domain (i.e., CC data) and in-domain (i.e., WMT19 En-De news domain), respectively. The word frequency is normalized and reported in log-scale.}
    \label{fig:distribution}
\end{figure}

\paragraph{Lexical Distribution in Training Data.}
Inspired by lexicon distribution analysis~\cite{ding2021understanding}, we first plot the word distributions of English corpora from general domain (i.e., CC data) and in-domain (i.e., WMT19 En-De news domain) to study their difference at the lexicon level. The words are ranked according to their frequencies in the WMT19 En-De training data. As shown in Figure~\ref{fig:distribution}, we observe a clear difference between WMT news data and CC data in the long tail region, which is supposed to carry more domain-specific information. Accordingly, there will be a domain shift from pretraining to finetuning.

\begin{table}[t!]
\fontsize{10}{12}\selectfont
  \centering
  \begin{tabular}{c cc cc}
  \toprule
  \bf Set
  & \bf En$\Rightarrow$De
  & \bf De$\Rightarrow$En \\
  \midrule
   Source & 77.5 & 73.7 \\
   Target & 71.0 &  75.4\\
  \bottomrule
  \end{tabular}
  \caption{Ratio of sentences in WMT19 En-De test sets that are classified as WMT news domain.}
  \label{tab:domain-cls}
\end{table}

\paragraph{Domain Classifier for Test Data.}
We further demonstrate that the test data also follows a consistent domain as the training data. To distinguish general domain and in-domain, we build a domain classifier based on the WMT19 En-De training data and the CC data.
We select a subset from the WMT training data with some trusted data~\cite{wang2018denoising,jiao2020data,jiao2022data2}, which includes 22404 sample from WMT newstest2010-2017 {(see Appendix~\ref{sec:domain-classifier} for details).}
Specifically, we select 1.0M samples from the WMT training data and the CC data, respectively, to train the domain classifier.
The newstest2018 is combined with an equally sized subset of CC data for validation. We adopt the domain classifier to classify each sample in the test sets of WMT19 En-De. As shown in Table~\ref{tab:domain-cls}, most of the sentences (e.g., 70\% - 80\%) are recognized as WMT news domain, which demonstrates the domain consistency between the training data and test data in the downstream tasks.

\subsubsection{Objective Discrepancy}
\label{sec:obj-discrepancy}

The learning objective discrepancy between Seq2Seq pretraining and NMT training is that NMT learns to translate a sentence from one language to another, while Seq2Seq pretraining learns to reconstruct the input sentence~\cite{Liu:2021:ACL}.
In this section, we study the side effects of the objective discrepancy by evaluating the predicting behaviors that are highly affected by the learning objective.

\begin{figure}[t!]
    \centering
    \subfloat[Reference]{
    \includegraphics[height=0.26\textwidth]{Figures/uncertainty-ende-small-ref.pdf}}
    \hfill
    \subfloat[Distractor]{
    \includegraphics[height=0.26\textwidth]{Figures/uncertainty-ende-small-dis.pdf}}
    \caption{Per-token generation probability on the test set of WMT19 En$\Rightarrow$De~(S). Higher probabilities are expected for the groundtruth references (a), and lower probabilities are expected for the distractors (b).}
    \label{fig:model-uncert}
\end{figure}

\paragraph{Model Uncertainty.}
We follow~\newcite{ott2018analyzing} to analyze the model's uncertainty by computing the average probability at each time step across a set of sentence pairs.
To evaluate the capability of LM modeling on the target language, we also follow~\newcite{wang2020exposure} to consider a set of ``distractor'' translations, which are random sentences from the CC data that match the corresponding reference translation in length.
Figure~\ref{fig:model-uncert} plots model uncertainties for both references ($Y$) and distractors ($\hat{Y}$).
We find that jointly pretraining decoder significantly improves model certainty after the first few time steps (Figure~\ref{fig:model-uncert}a).
As for the distractors, pretraining encoder only results in certainties even lower than training from scratch (Figure~\ref{fig:model-uncert}b), which suggests that the corresponding NMT model is more dominated by the source context. It reconfirms the finding in our human evaluation (Table~\ref{tab:mbart-adequacy}). In contrast, jointly pretraining decoder leads to a significant improvement of certainties, suggesting that the pretrained decoder tends to induce the {\bf over-estimation issue} of NMT models. A possible reason is that Seq2Seq pretraining does not establish the connection between languages, such that its strong capability of LM modeling still recognizes the distractor as a valid target sentence even though it is mismatched with the source sentence in semantics.

\paragraph{Hallucination under Perturbation.}
One translation problem associated with over-estimation is hallucination~\cite{wang2020exposure}, where NMT models generate fluent translation but is unrelated to the input. In this section, we follow~\newcite{lee2018hallucinations} to evaluate the model's tendency of generating hallucination under noisy input, to which NMT models are highly sensitive~\cite{belinkov2018synthetic}.
Specifically, we employ two different perturbation strategies: (1) First position insertion (FPI) that inserts a single additional input token into the source sequence, which can completely divorce the translation from the input sentence~\cite{lee2018hallucinations}.
(2) Random span masking (RSM) that simulates the noisy input in the Seq2Seq pretraining of mBART~\cite{Liu2020mbart}.
We follow ~\newcite{lee2018hallucinations} to count a translation as hallucination under perturbation (\textsc{HuP}) when: (1) BLEU between reference sentence and translation of unperturbed sentence is bigger than 5 and (2) BLEU between the translation of perturbed sentence and the translation of unperturbed sentence is lower than 3. We calculate the percentage of hallucination as the \textsc{HuP} score. Table~\ref{tab:mbart-halluc} lists the BLEU change and \textsc{HuP} score for the perturbed inputs. As expected, jointly pretraining decoder is less robust to perturbed inputs (more decline of BLEU scores), and produces more hallucinations than the other two model variants.

\begin{table}[t!]
\fontsize{10}{12}\selectfont
\centering
\begin{tabular}{cc rr rr}
\toprule
\multicolumn{2}{c}{\bf Pretrain}
& \multicolumn{2}{c}{\bf FPI (\%)}
& \multicolumn{2}{c}{\bf RSM (\%)}\\
\cmidrule(lr){1-2}  \cmidrule(lr){3-4} \cmidrule(lr){5-6}
\em {Enc} & \em {Dec}
& \em  $\triangle_{\mathrm{BLEU}}$ & \em \textsc{HuP}
& \em $\triangle_{\mathrm{BLEU}}$ & \em \textsc{HuP} \\
\midrule
\texttimes  &   \texttimes & -1.3 & 0.5 & -8.8 & 2.4  \\
\checkmark  &   \texttimes & -0.3 & 0.5 & -8.3 & 0.5  \\
\checkmark  &   \checkmark & \bf -3.2 & \bf 7.8 & \bf -17.8 & \bf 15.5 \\
\bottomrule
\end{tabular}
\caption{BLEU change of model performance under perturbed inputs over the standard inputs, and hallucinations under perturbation (HUP) score.}
\label{tab:mbart-halluc}
\end{table}

\begin{table}[t!]
\fontsize{10}{12}\selectfont
\centering
\begin{tabular}{cc cc cc}
\toprule
\multicolumn{2}{c}{\bf Pretrain}
& \multicolumn{2}{c}{\bf BLEU}
& \multicolumn{2}{c}{\bf Copy (\%)}\\
\cmidrule(lr){1-2}  \cmidrule(lr){3-4} \cmidrule(lr){5-6}
\em {Enc} & \em {Dec}
& 5 & 100 & 5 & 100\\
\midrule
\texttimes  &   \texttimes & 26.7 & \cellcolor{zgreen!5}26.6 & 12.9 & 13.9 \\
\checkmark  &   \texttimes & 31.7 & \cellcolor{zgreen!5}31.6 & 12.7 & 12.9 \\
\checkmark  &   \checkmark & 35.3 & \cellcolor{zgreen!90}33.5 & \bf 13.2 & \bf 19.4 \\
\bottomrule
\end{tabular}
\caption{Beam search degradation and ratio of copying tokens in translation outputs.}
\label{tab:mbart-BSD}
\end{table}

\paragraph{Beam Search Problem.}
One commonly-cited weakness of NMT model is the beam search problem, where the model performance declines as beam size increases~\cite{Tu:2017:AAAI}.
Previous studies demonstrate that over-estimation is an important reason for the beam search problem~\cite{ott2018analyzing,cohen2019empirical}.
We revisit this problem for NMT models with Seq2Seq pretraining, as shown in Table~\ref{tab:mbart-BSD}.
We also list the ratio of copying tokens in translation outputs (i.e., directly copy source words to target side without translation) for different beam sizes, which has been shown as a side effect of Seq2Seq pretraining models~\cite{Liu:2021:ACL}.
As seen, jointly pretraining decoder suffers from more serious beam search degradation problem, which reconfirms the connection between beam search problem and over-estimation.
In addition, larger beam size introduces more copying tokens than the other model variants (i.e., 19.4 vs. 13.9, 12.9), which also links copying behaviors associated with Seq2Seq pretraining to the beam search problem.

\section{Improving Seq2Seq Pretraining}
\label{sec:approach}

\subsection{Approach}

To bridge the above gaps between Seq2Seq pretraining and finetuning, we introduce {\em in-domain pretraining} and {\em input adaptation} to improve the translation quality and model robustness.

\paragraph{In-Domain Pretraining.}
To bridge the domain gap, we propose to continue the training of mBART~\cite{Liu2020mbart} on the in-domain monolingual data. Specifically, we first remove
spans of text and replace them with a mask token. We mask 35\% of the words in each sentence by random sampling a span length according to a Poisson distribution ($\lambda=3.5$). We also permute
the order of sentences within each instance. The training objective is to reconstruct the original sentence at the target side.
We expect the in-domain pretraining to reduce the domain shift by re-pretraining on the in-domain data, which is more similar in data distribution with the downstream translation tasks.

\paragraph{Input Adaptation in Finetuning.}
To bridge the objective gap and improve the robustness of models, we propose to add noises (e.g., mask, delete, permute) to the source sentences during finetuning, and keep target sentences as original ones. Empirically, we add noises to 10\% of the words in each source sentence, and combine the noisy data with the clean data by the ratio of 1:9, which are used to finetune the pretraining model.
We expect the introduction of perturbed inputs in finetuning can help to better transfer the knowledge from pretrained model to the finetuned model, thus alleviate over-estimation and improve the model robustness.

\begin{table*}[t!]
  \centering
  \begin{tabular}{l cr cr cr cr}
  \toprule
  \multirow{2}{*}{\bf Approach}
  & \multicolumn{2}{c}{\bf W19 En$\Rightarrow$De}
  & \multicolumn{2}{c}{\bf \small W19 En$\Rightarrow$De (S)}
  & \multicolumn{2}{c}{\bf W16 En$\Rightarrow$Ro}
  & \multicolumn{2}{c}{\bf I17 En$\Rightarrow$Fr} \\
  \cmidrule(lr){2-3} \cmidrule(lr){4-9}
 & BLEU & \textsc{HuP} &  BLEU & \textsc{HuP} & BLEU & \textsc{HuP} &  BLEU & \textsc{HuP} \\
  \midrule
  Baseline   & 39.4 & 2.6  & 26.7 & 2.4  & 30.0 &  1.1 & 35.3 & 1.6 \\

  General   & 40.8 & 3.3  & 35.3 & 15.5  & 37.1 &  6.5 & 39.2 & 7.8 \\
  \hline
 ~~~+ Input Adapt & 40.8 & 2.7 &  35.6 &  5.7 & 37.2 & 2.4 & 39.4 & 1.5 \\
  ~~~+ In-Domain & \bf 42.2 & 9.2   & \bf 36.4 &  10.4  & \bf 38.0 &  8.2  & 39.9 &  5.5  \\
  ~~~~~~+ Input Adapt & 41.3  & 4.1 &  36.1  & 3.6 & 37.8 & 2.9 & \bf 40.1 & 3.0\\
  \bottomrule
  %
  \multirow{2}{*}{\bf Approach}
  & \multicolumn{2}{c}{\bf W19 De$\Rightarrow$En}
  & \multicolumn{2}{c}{\bf \small W19 De$\Rightarrow$En (S)}
  & \multicolumn{2}{c}{\bf W16 Ro$\Rightarrow$En}
  & \multicolumn{2}{c}{\bf I17 Fr$\Rightarrow$En} \\
  \cmidrule(lr){2-3} \cmidrule(lr){4-9}
 & BLEU & \textsc{HuP} &  BLEU & \textsc{HuP} & BLEU & \textsc{HuP} &  BLEU & \textsc{HuP} \\
  \midrule
  Baseline &  40.1  & 2.8  &27.1   &  1.3 &  29.6 & 1.3   & 35.1 & 1.7 \\

  General   & \bf 41.4 & 7.7 & 35.7 &  4.9  &  37.4 &  6.0 &  40.2 &  4.7 \\
  \hline
  ~~~+ Input Adapt & 41.2 & 2.6 &  35.9 &  2.8 & 37.1 & 3.5 & 40.7 & 2.5 \\
  ~~~+ In-Domain & 41.3  &  8.2  & \bf 36.9 &    7.4  & \bf 38.1 & 7.7 & \bf 41.1  & 4.2\\
  ~~~~~~+ Input Adapt & \bf 41.4  &3.1 &  36.8  & 2.9 & 37.9 & 3.9 & 41.0 & 1.7\\
  \bottomrule
  \end{tabular}
  \caption{
  BLEU and \textsc{HuP} scores of our approaches for downstream translation tasks.}
  \label{tab:main-results}
\end{table*}

\subsection{Experimental Results}

\paragraph{Main Results on Translation Performance and Robustness.}
The main results are listed in Table~\ref{tab:main-results}. We report the results of input adaptation, in-domain pretraining, and the combination of these two approaches, respectively. For input adaptation, it achieves comparable translation quality as the general domain pretrained model and significantly reduces the ratio of \textsc{HuP}, indicating the enhancement of model robustness.
In-domain pretraining generally improves the translation quality but does not make the model more robust. On the contrary, it may increase the ratio of \textsc{HuP} in some cases (e.g., En$\Rightarrow$Ro 5.6 vs. 8.2). Conducting input adaptation right after in-domain pretraining will combine the advantages of these two approaches, and improve both the translation quality and model robustness.
{The effectiveness of our approaches, especially input adaptation, is more significant when evaluated with multiple references, as shown in Table~\ref{tab:multi-ref-impr}. }

\begin{table}[t!]
\setlength{\tabcolsep}{4pt}
\fontsize{10}{12}\selectfont
\centering
  \begin{tabular}{l cc cc}
  \toprule
  \bf Approach
  & \multicolumn{2}{c}{\bf W19 En-De}
  & \multicolumn{2}{c}{\bf \small W19 En-De~(S)} \\
  \cmidrule(lr){2-3}  \cmidrule(lr){4-5}
&   BLEU  &  $\triangle$ &   BLEU    &   $\triangle$  \\
  \midrule
   Baseline & 75.7  & - &  52.3 & - \\
   General & 79.1 & +3.4  &  69.1 & +16.8 \\
   ~~~+ Input Adapt &  79.2 & +3.5 &  71.7 & +19.4\\
   ~~~+ In-Domain & \bf 80.1 & \bf +4.4 &  73.7 & +21.4\\
   ~~~~~~+ Input Adapt & 79.8 &  +4.1 & \bf 75.6 & \bf +23.3\\
  \bottomrule
  \end{tabular}
  \caption{BLEU scores with multiple references.}
  \label{tab:multi-ref-impr}
\end{table}

\paragraph{In-Domain Only.}
Given the promising performance of in-domain pretraining, we investigate whether pretraining on in-domain data only can also obtain significant improvement.
We report the results in Table~\ref{tab:indomain-only}. We can observe that pretraining solely on the in-domain data can improve the translation performance noticeably over the models without pretraining. However, the improvement is less competitive than the pretrained mBART25 (e.g., En$\Rightarrow$Ro: 36.1 v.s. 37.1 in Table~\ref{tab:main-results}), which may result from the much larger scale of multilingual data used in general pretraining.

\begin{table}[t!]
\setlength{\tabcolsep}{4pt}
\fontsize{10}{12}\selectfont
  \centering
  \begin{tabular}{l cc cc}
  \toprule
  \bf Approach
  & \multicolumn{2}{c}{\bf\small W19 En-De (S)}
  & \multicolumn{2}{c}{\bf W16 En-Ro}\\
  \cmidrule(lr){2-3} \cmidrule(lr){4-5}
  &  $\Rightarrow$  &   $\Leftarrow$  &  $\Rightarrow$  & $\Leftarrow$ \\
  \midrule
   Baseline & 26.7 & 27.1 & 30.0 & 29.6 \\
   In-Domain & 35.2 & 35.7 & 36.1 & 36.3 \\
  \bottomrule
  \end{tabular}
  \caption{BLEU scores of in-domain pretraining only.}
  \label{tab:indomain-only}
\end{table}

\subsection{Analysis}
We provide some insights into how our approach improves model performance over general pretraining. We report results on WMT19 En$\Rightarrow$De test set using small-scale data.

\paragraph{Narrowing Domain Gap.}
Since the difference of lexical distribution between general domain and in-domain data mainly lies in the long tail region (see Figure~\ref{fig:distribution}), we study how our approach performs on low-frequency words.
Specifically, we calculate the word accuracy of the translation outputs for WMT19 En-De~(S) by the \texttt{compare-mt}\footnote{\url{https://github.com/neulab/compare-mt}} tool. We follow previous studies~\cite{Wang:2021:ACL,Jiao2021SelfTrainingSW} to divide words into three categories based on their frequency in the bilingual data, including High: the most {3,000} frequent words; Medium: the most {3,001}-{12,000} frequent words; Low: the other words.
Table~\ref{tab:linguistic-freq} lists the results.
The improvements on low-frequency words are the major reason for the performance gains of in-domain pretraining, where it outperforms general pretraining on the translation accuracy of low/medium/high- frequency words by 1.7, 0.0, and 0.7 BLEU scores, respectively.
These findings confirm our hypothesis that in-domain pretraining can narrow the domain gap with in-domain data, which is more similar in the lexical distribution as the test sets.

\begin{table}[t!]
\fontsize{10}{12}\selectfont
  \centering
  \begin{tabular}{l ccc}
  \toprule
  \multirow{2}{*}{\bf Approach}  &  \multicolumn{3}{c}{\bf Frequency} \\
  \cmidrule(lr){2-4}
   & \it Low  & \it Med &  \it High \\
  \midrule
   Baseline  & 36.8 & 45.3 & 57.5  \\
   General  & 44.5 & 54.3 & 64.2 \\
   ~~+ In-Domain & 46.2 & 54.3 & 64.9 \\
  \bottomrule
  \end{tabular}
  \caption{F-measures of word prediction for different frequencies that are calculated in the bilingual data.}
  \label{tab:linguistic-freq}
\end{table}

\paragraph{Alleviating Over-Estimation.}
Figure~\ref{fig:model-uncert-impr} shows the impact of our approach on model uncertainty.
Clearly, our approach successfully alleviates the over-estimation issue of general pretraining in both the groundtruth and distractor scenarios.

\paragraph{Mitigating Beam Search Degradation.}
We recap the beam search degradation problem with the application of our approaches in Table~\ref{tab:mbart-BSD-impr}.
The input adaptation approach can noticeably reduce the performance decline when using a larger beam size (e.g., from -1.8 to -0.9), partially due to a reduction of copying tokens in generated translations (e.g., from 19.4\% to 15.3\%). Although in-domain pretraining does not alleviate the beam search degradation problem, it can be combined with input adaptation to build a well-performing NMT system.

\section{Related Work}

\paragraph{Pretraining for NMT.}
Previous pretraining approaches for NMT generally focus on how to effectively integrate pretrained BERT~\cite{Devlin2019BERT} or GPT~\cite{radford2019GPT2} to NMT models. For example,~\newcite{yang2020towards} propose a concerted training framework, and~\newcite{weng2020acquiring} propose a dynamic fusion mechanism and a distillation paradigm to acquire knowledge from BERT and GPT.
In this work, we aim to provide a better understanding of how Seq2Seq pretraining model works for NMT, and propose a simple and effective approach to improve model performance based on these observations.

\paragraph{Intermediate Pretraining.}
Our in-domain pretraining approach is related to recent successes on intermediate pretraining and intermediate task selection in NLU tasks.
For example, \newcite{ye2021influence} investigate the influence of masking policies in intermediate pretraining. \newcite{poth2021pre} explore to select tasks for intermediate pretraining. Closely related to our work, \newcite{gururangan2020dont} propose to continue the pretraining of \textsc{RoBERTa}~\cite{liu2019roberta} on task-specific data.
Inspired by these findings, we employ in-domain pretraining to narrow the domain gap between general Seq2Seq pretraining and NMT training. We also show the necessity of target-side monolingual data on in-domain pretraining (see Appendix~\ref{sec:indomain-langs}), which has not been studied in previous works of in-domain pretraining.

\begin{figure}[t]
    \centering
    \subfloat[Reference]{
    \includegraphics[height=0.26\textwidth]{Figures/uncertainty-ende-small-impr-ref.pdf}}
    \hfill
    \subfloat[Distractor]{
    \includegraphics[height=0.26\textwidth]{Figures/uncertainty-ende-small-impr-dis.pdf}}
    \caption{Per-token generation probability on WMT19 En$\Rightarrow$De~(S) test set when adopting our approaches.}
    \label{fig:model-uncert-impr}
\end{figure}

\begin{table}[t]
\centering
\fontsize{10}{12}\selectfont
\begin{tabular}{l cc ll }
\toprule
\multirow{2}{*}{\bf Approach}
& \multicolumn{2}{c}{\bf BLEU}
& \multicolumn{2}{c}{\bf Copy (\%)}\\
\cmidrule(lr){2-3}  \cmidrule(lr){4-5}
& 5 & 100 & 5  & 100 \\
\midrule
General & 35.3  & \cellcolor{zgreen!90}33.5 & \bf 13.2  & 19.4 \\
~~+ Input Adapt & 35.6  & \cellcolor{zgreen!45}34.7 & 12.5$^\downarrow$  & 15.3$^\downarrow$ \\
~~+ In-Domain & 36.4  & \cellcolor{zgreen!100}33.9 & 12.9  & \bf 19.8 \\
~~~~+ Input Adapt & 36.1 & \cellcolor{zgreen!55}35.0 & 12.6$^\downarrow$ & 15.6$^\downarrow$ \\
\bottomrule
\end{tabular}
\caption{Beam search degradation and ``copy'' translations when adopting our approaches.}
\label{tab:mbart-BSD-impr}
\end{table}

\section{Conclusion}

In this paper we provide a better understanding of Seq2Seq pretraining for NMT by showing both the benefits and side effects. We propose simple and effective approaches to remedy the side effects by bridging the gaps between Seq2Seq pretraining and NMT finetuning, which further improves translation performance and model robustness.
Future directions include validating our findings on more Seq2Seq pretraining models and language pairs.

redpajama_set_name: RedPajamaArXiv


[15-04-2019 07:26:39] - Johnny Gaudreau from Ottawa Senators is back from Lacerated Right Calf Injury.
[14-12-2018 23:19:21] - Johnny Gaudreau from Ottawa Senators has scored a Hat Trick!
[08-11-2018 12:31:05] - Johnny Gaudreau ratings were not updated!
[19-06-2018 20:39:37] - Johnny Gaudreau ratings were not updated!
[22-06-2018 08:09:37] - Johnny Gaudreau ratings were not updated!
[29-06-2018 06:36:21] - Johnny Gaudreau ratings were not updated!
[05-05-2018 09:10:32] - Johnny Gaudreau ratings were not updated!
[05-05-2018 07:49:18] - Johnny Gaudreau ratings were not updated!
[04-05-2018 23:28:58] - Johnny Gaudreau ratings were not updated!
[11-11-2017 09:06:14] - Johnny Gaudreau from Ottawa Senators is back from Sore Right Knee Injury.
[11-11-2017 09:06:11] - Game 242 - Johnny Gaudreau from Ottawa Senators is injured (Sore Right Knee) and is out for 5 days.
[29-06-2017 08:03:44] - Johnny Gaudreau ratings were not updated!
[23-06-2017 22:49:43] - Johnny Gaudreau ratings were not updated!
[04-06-2017 22:41:59] - Johnny Gaudreau ratings were not updated!
[14-08-2016 17:27:49] - Johnny Gaudreau ratings were not updated!
"Johnny Hockey" is a diminutive, offensively dominant winger with an incredibly high level of skill. Possesses the elusiveness to avoid being taken out, and the creativity to start and finish plays. Exhibits incredible puck control, strength on his skates, stickhandling ability, and a very good forehand and backhand shot which is off of his stick in the blink of an eye. All-in-all, the prototypical offensive winger that can put up points, and be relied upon to create scoring chances, whenever he is on the ice.


Barmy about burgers? Have a love for lobsters? Craving a career in the restaurant industry? If so, we want to hear from you!
We are so proud of the rewarding work environments that we've created and our ability to offer the same development opportunities for all. Our teams are made up of 35 different nationalities, from 6 continents all who benefit from our supportive and inclusive company culture. It is this diverse workforce that makes Burger & Lobster such a special company to work for.
Explore our current vacancies below and click the links to apply today.


text:
One Person's Creepy is Another Person's Charming
By Dustin Rowles | Pajiba Love | December 21, 2009 |
The Captain of the Pajiba Love Boat isn't here today, having reportedly drowned in the Philly snow over the weekend. She'll return when certain advancements in cryogenics are made. Or tomorrow. Whichever comes first. Meanwhile, you're stuck with me.
Our own Mr. Carlson takes a look at the best and worst Christmas specials of all time, careful to note that those Rankin-Bass specials will give you psychic scars that will last a lifetime. (Hair Balls)
Tron: Legacy opens in 361 days. While you're waiting, check out the Tron Light Cycle. Geek. (FilmDrunk)
Friend of the site, The Second Pass' John Williams, discusses the books he most enjoyed in 2009. (The Millions)
Are you like me? Have you been wondering where Rene Russo disappeared off to about five years ago? Looks like she's resurfacing. On Thor. (NewsinFilm)
Here are the ten best music documentaries of the Aughts from the always brilliant (The Playlist)
My favorite part about the new teaser poster for Cameron Diaz and Tom Cruise's next movie, Night and Day is that Cameron Diaz and Tom Cruise aren't in it. (Agent Bedhead)
EW's worst book of the year goes to Spencer and Heidi. Meanwhile, EW's worst magazine of the year goes to EW. (Litelysalted)
Johnny Knoxville says that, for the next Jackass movie (in 3D), he's going to take all that Avatar technology and shove it up Steve-O's ass. (Hitflix)
I'm hoping to catch the new web series "Coma, Period," but maybe if someone else tells me if it's worth watching, I'd be more inclined to do so. (Coma, Period)
Hey! Look here! The Associated Press named Taylor Swift Entertainer of the Year. In other news, the Associated Press is a joke. (AP)
In news you couldn't possibly give a shit about, I was knocked out of my fantasy football playoffs this weekend, which is a tiny humiliation compared to this Oakland Raider, who lost his fucking pants during a game. (YouTube)
A half-day in a make-up chair and some Photoshop work, and you, too, could like like a cast member of "Jersey Shore." If they could do it to Alyssa Milano, they could do it to anyone.
Alyssa Milano's Evolution: Jersey Shore from Alyssa Milano
General Apathy and Major Boredom Singing ...
The Top 20 Films of the Decade -- Naysayer's Edition

redpajama_set_name: RedPajamaCommonCrawl


Use panels for a built-in appearance.
Panels are used to make appliances appear as if they are built into the cabinets. They are made from the same species of wood as the existing cabinets and are cut to fit the appliances' height and width. They install to the wall and to the floor alongside the appliance and can be removed for easy access when needed. Appliance panels have hardwood styles glued to the front to match the hardwood in the kitchen. You can make an appliance panel in about an hour.
Measure the height and width of the appliance. For stoves, measure from the floor to the top of the burners. For refrigerators and dishwashers, measure to the top. Subtract 1 1/2-inches from both measurements. Cut a piece of plywood to that measurement, using a table saw. For example, if your measurement is 22-by-36 inches, cut the plywood at 20 1/2-by-34 1/2 inches.
Set a miter saw at 45 degrees. Miter all four pieces of hardwood on both ends.
Run a bead of glue along the back of each mitered piece, 3/8-inch up from the bottom edge. Place the plywood panel flat on a worktable. Place all four mitered pieces of hardwood on their edges around the panel. Bring the mitered corners together and shoot pin nails into the hardwood 6 inches apart to secure the hardwood frame to the plywood. The panel should have a 3/4-inch lip.
Turn the panel over. Putty all the nail holes with a putty knife and wood dough. Sand the panel smooth using 100-grit sandpaper on a hand block. Stain and lacquer as needed.
Slide the appliance away from the wall. Stand the panel in place where you wish to install it. Predrill four holes evenly spaced through the back lip of the panel where it touches the wall, using a cordless drill and 3/16-inch drill bit. Predrill four holes through the bottom lip of the panel. Place 3-inch screws into the holes. Use the cordless drill to screw the panel tightly to the floor and the wall. Slide the appliance back into place.
If you can't hit a stud or brace along the back of the panel, use drywall inserts that expand behind the drywall to secure the panel to the wall.
Shaddy, Wade. "How to Make Appliance Panels." Home Guides | SF Gate, http://homeguides.sfgate.com/make-appliance-panels-20110.html. Accessed 26 April 2019.


text:
Q: 3D Hexagonal Grid CSS based not displaying as wanted The goal of this is to make a dynamic Hexagonal Grid using CSS, so that when a new div with certain class (eg. 'hexes') is added to the view the grid would automatically reorganize the hexes in a circular or rectangular way.
So far nothing of this has been done, what i have is a grid with lots of hexes, with some animations, they're ordered by columns of 4 hex-boxes, we can add rows to the columns, or we can add columns with rows, but they aren't responsive either self-ordered.
Here's a snippet of the mentioned above

body { background-color: #171d25; }

.container {
  position: relative;
  -webkit-transition: all 0.3s ease-out;
  transition: all 0.3s ease-out;
  height: 500px;
  -webkit-perspective: 700px;
  perspective: 700px;
  z-index: 10;
  overflow: hidden;
}

.container .honeycomb {
  display: block;
  position: absolute;
  margin-left: -465px;
  padding-left: 18px;
  left: 50%;
  text-align: center;
  height: 450px;
  width: 930px;
  -ms-transform: rotateX(45deg) rotateY(0deg);
  -webkit-transform: rotateX(45deg) rotateY(0deg);
  transform: rotateX(45deg) rotateY(0deg);
  -webkit-transform-style: preserve-3d;
  transform-style: preserve-3d;
  -ms-transform-origin: center 65%;
  -webkit-transform-origin: center 65%;
  transform-origin: center 65%;
  z-index: 5;
}

.container .honeycomb .column {
  display: block;
  float: left;
  width: 62px;
}

.container .honeycomb .column:nth-child(odd) { margin-top: 36px; }

.container .honeycomb .hex {
  display: block;
  position: relative;
  float: left;
  margin: 0;
  height: 72px;
  width: 62px;
  color: #fff;
  cursor: pointer;
  text-decoration: none;
  -webkit-transform-style: preserve-3d;
  transform-style: preserve-3d;
  -webkit-transition: all 0.3s ease-out;
  transition: all 0.3s ease-out;
 filter: progid:DXImageTransform.Microsoft.Alpha(Opacity=50);
  opacity: 0.5;
}

.container .honeycomb .hex:hover {
  opacity: 1;
  -ms-transform: translateZ(20px);
  -webkit-transform: translateZ(20px);
  transform: translateZ(20px);
  -webkit-transition-duration: 200ms;
  transition-duration: 200ms;
}

.container .honeycomb .hex:hover .content {
 filter: progid:DXImageTransform.Microsoft.Alpha(enabled=false);
  opacity: 1;
  -ms-transform: rotateX(-90deg) translateZ(-40px) translateY(-50px);
  -webkit-transform: rotateX(-90deg) translateZ(-40px) translateY(-50px);
  transform: rotateX(-90deg) translateZ(-40px) translateY(-50px);
}

.container .honeycomb .hex .wrapper {
  display: block;
  position: absolute;
  top: 0;
  left: 0;
  height: 100%;
  width: 100%;
}

.container .honeycomb .hex .wrapper .color-1 { background: #ffbc06; }

.container .honeycomb .hex .wrapper .color-1:before { border-right-color: #ffbc06; }

.container .honeycomb .hex .wrapper .color-1:after { border-left-color: #ffbc06; }

.container .honeycomb .hex .content {
  display: block;
  position: absolute;
  top: 0;
  left: -80px;
  text-align: center;
  width: 200px;
 filter: progid:DXImageTransform.Microsoft.Alpha(Opacity=0);
  opacity: 0;
  -moz-transition: all 0.3s ease-out 0.1s;
  -o-transition: all 0.3s ease-out 0.1s;
  -webkit-transition: all 0.3s ease-out;
  -webkit-transition-delay: 0.1s;
  -webkit-transition: all 0.3s ease-out 0.1s;
  transition: all 0.3s ease-out 0.1s;
  -ms-transform: rotateX(-90deg) translateZ(-40px) translateY(-40px);
  -webkit-transform: rotateX(-90deg) translateZ(-40px) translateY(-40px);
  transform: rotateX(-90deg) translateZ(-40px) translateY(-40px);
  pointer-events: none;
  -webkit-font-smoothing: antialiased;
}

.container .honeycomb .hex .content strong {
  display: block;
  font: 600 26px/1.1 verdana, sans-serif;
  overflow: hidden;
  text-overflow: ellipsis;
  white-space: nowrap;
  text-shadow: -1px -1px 2px rgba(0, 0, 0, 0.8);
}

.container .honeycomb .hex .content small {
  display: block;
  font: 15px/1.1 verdana, sans-serif;
}

.container .honeycomb .shadows {
  display: block;
  position: absolute;
  top: -15%;
  left: -15%;
  height: 130%;
  width: 130%;
  background-image: -webkit-radial-gradient(closest-side, rgba(23, 29, 37, 0), #171d25);
  background-image: radial-gradient(closest-side, rgba(23, 29, 37, 0), #171d25);
 filter: progid:DXImageTransform.Microsoft.Alpha(Opacity=90);
  opacity: 0.9;
  -ms-transform: translateZ(10px);
  -webkit-transform: translateZ(10px);
  transform: translateZ(10px);
  pointer-events: none;
}

.hexagon {
  display: block;
  position: absolute;
  width: 36px;
  height: 62px;
  background-color: #ffffff;
}

.hexagon:before,
.hexagon:after {
  content: "";
  position: relative;
  float: left;
  border-top: 31px solid transparent;
  border-bottom: 31px solid transparent;
}

.hexagon:before {
  border-right: 18px solid #ffffff;
  left: -18px;
}

.hexagon:after {
  border-left: 18px solid #ffffff;
  right: -18px;
}

.hidden { display: none; }

3D Hexagon Grid Layout Demo

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

       CSSScript i'm a hexagon

Now if i would like half of the hexes i would get rid off of seven div with class 'column'... but the hexes won't expand neither get centered.
Here's an attempt to make them bigger and display them centered, basically what was done here is make bigger the square, adjust some left and right values, nothing solid...
Someone told me to use @media queries, but i haven't worked with them so much... What's needed is to make the grid resposive, and also dynamic, the purpose is to make it work with AngularJS using an ng-repeat, so that we won't know the number of options, and it will order them to make them view nice. Also would like to know how to do some other stuff like insert an image in the middle of the hex, display an image on top of the name of the hex, and maybe display some info with an animation...

A: The only solution available yet is indeed using media queries. The concept is that you know the width of the hexagons, you use :nth-child() selector to push every second row instead of wrapping the hexagons in rows, and just make them inline-block, and then setup all the breakpoints with media query based on your hexagons width and only change the value inside :nth-child() when the viewport width changes.
Example:

The formaula is:
for(i = 2; i < .hex(length); i++){
  @media(min-width: i * .hex(width) ) and (max-width: (i * 2 * .hex(width)) - 1){
    .hex:nth-child(((2*i)-1)n+(i+1)){
      margin-left:.hex(width)/2;
    }
  }
}

This works well for horizontal (pointy) hex grids, but the idea can be easily adapted for vertical (flat) hex grids changing a bit the formula above.
Here is the fiddle: https://jsfiddle.net/q2my8vcn/
(Already tweaked some, still not perfect but you can see the basic idea quite well in action.)

redpajama_set_name: RedPajamaStackExchange


text:
Older Workers Find Age Discrimination Built Right Into Some Job Websites The sites screen out older job seekers by limiting the dates you can fill in forms, an investigation by the Illinois attorney general found. But other sites say they make a point of fairness.
Older Workers Find Age Discrimination Built Right Into Some Job Websites
March 28, 20179:12 AM ET
Heard on All Things Considered
Ina Jaffe
Older adults have been excluded from some websites that post jobs, according to Illinois Attorney General Lisa Madigan. Jonathan McHugh/Getty Images/Ikon Images hide caption
Jonathan McHugh/Getty Images/Ikon Images
Older adults have been excluded from some websites that post jobs, according to Illinois Attorney General Lisa Madigan.
If you're looking for work, you might start with one of those websites that posts jobs. But if you're an older adult looking for work, you might have found yourself excluded from some of the features on those sites.
Illinois Attorney General Lisa Madigan opened an investigation after a 70-year-old man called her office and complained that he'd been unable to use a resume building tool on a job search site. (Read Madigan's letters requesting information from job website companies she's investigating.)
The problem was the drop-down menu that required you to select the year when you graduated or got your first job. Those dates only went back to 1980. Madigan's office contends that could exclude anyone over 52.
Madigan's office contacted Monster.com, Ladders Inc., Beyond.com, Indeed.com (an NPR funder), CareerBuilder and Vault. Their sites were less restrictive, but they all had their limits. Some only excluded people older than 82, but Madigan says that's still discrimination. The way these job search sites should work, she says, is that "anybody who's alive and wants to look for a job would be able to, and be able to put in accurate information."
Responding to NPR's request for comment, some of the job search sites said the date limit was inadvertent and that they have corrected it. Others wrote about special efforts they make to assist and advise older job hunters. Their full responses are at the end of this article.
Madigan also wants to see the companies' internal documents to see if the exclusions were deliberate or accidental. At this writing, she had received written responses from only Beyond and Indeed, but none of the documents requested.
"Our goal is to fix the problem," says Madigan. "Our goal is not to file a lawsuit."
Some organizations are trying to "fix the problem" in other ways. The AARP has an Employer Pledge Program, where companies promise equal opportunity for all workers and job seekers regardless of age.
Kerry Hannon is the AARP's work and jobs expert and the author of 10 books on work and finance in later life. She says that so many employees who are over 60 tell her that "they look across the table and there is this idea that the employer sees their expiration date." The Employer Pledge Program is important, she says, because "age discrimination is alive and well in the workplace" and AARP members want to know how they can avoid it.
There are also online job search sites specifically geared toward older workers. One of them, RetirementJobs.com, posts open positions at any company that wants to place a listing. But the site also certifies some employers as "age friendly." Tim Driver, the founder and CEO, says an employer is awarded that designation only after his staff investigates the company's practices and culture.
"There's over 100 big American employers now that have gone through the [age friendly] program," says Driver. Those include Fidelity and Wells Fargo and "even government organizations, like the TSA or the Veterans Administration."
Too Much Experience To Be Hired? Some Older Americans Face Age Bias
Asian Last Names Lead To Fewer Job Interviews, Still
But job-seekers don't have to take Driver's word for it. RetirementJobs.com is the only job search site for older workers that lets members post comments describing their experiences with specific companies, Yelp-style.
Driver says they've just started doing this in the past few months. "We wanted to know, are these companies truly walking the talk?" he says.
Well, not always. There are dozens of comments on the site about companies that force out older workers or exclude older job hunters in favor of "recent graduates." On the other hand there are also plenty of five-star reviews, most brimming with gratitude for companies that treat older workers with respect and don't hold their longer experience against them.
Below are responses from job search sites to NPR regarding the Illinois attorney general's investigation.
Monster does not have a comment regarding the Illinois Attorney General's investigation at this time.
CareerBuilder is committed to helping workers of all ages find job opportunities, and has fixed this unfortunate oversight.
TheLadders.com
Ladders has not received a letter from Illinois Attorney General Lisa Madigan to date, nor do we participate in these practices. Our site does not restrict the dates on resumes that people submit to us. In fact, to combat age discrimination we do not ask for a year of graduation from college. Our optional work experience section limits your experience to the past 50 years. We have spoken publically about this topic and feel very passionate about combatting age discrimination in the workforce. What we have shared with job seekers is our belief that to effectively market yourself it is best not to discuss work experience from more than twenty years ago. You can find that information here: https://www.theladders.com/p/12911/y2k-bug-resume
Beyond.com
Age discrimination is a pervasive issue in the hiring process, especially now that many employees are retiring later or coming back to the work force later in life, and we have actively counseled job candidates of an older age to ensure they are positioning themselves in the best manner. It is commonly recognized in this industry that it's best to leave age references off of one's resume. Career advocates emphasize that employers are mainly interested in current experience, regardless of the age of the candidate. Experience older than 20 years is generally not considered relevant. Online resume-building tools that force applicants to enter years for degree programs or work experience allow an indirect method for an employer to gauge the age of the applicant. We take a tremendous amount of care to help our members and employers carefully consider all information they put forth during the job search process to avoid any conscious or unconscious bias. Thus, the dropdown fields for years of employment or education are optional in the Career Portfolio. Regardless, the member can include all dates they desire in their resume, which is uploaded to our site. The Career Portfolio is simply a free service to allow a member to publicly market themselves in a creative way through a url. It is not a primary way to find a job, nor does it replace a resume. After receiving the letter from the Illinois Attorney General, we have since changed the dropdown menu for education or employment dates to go back much further. This will hopefully satisfy the concern of the Illinois Attorney General that all job seekers will have the opportunity to offer this information to potential employers if they wish.
Indeed.com [Indeed.com is among NPR's financial supporters]
Indeed's mission is to help people get jobs, and we strongly believe that age should not be a factor in evaluation of employment. On Indeed, anyone can upload a resume with any dates, and users can create a resume with drop down dates that go back to 1900.
Indeed has always allowed people of all ages to search for jobs, create resumes, post resumes and apply for jobs through our site. Beyond being able to post a resume, our free resume creation product drop down menu went back to 1956. This did not prevent anyone from manually noting an earlier date on a resume, but we did extend that menu to 1900 after hearing of the concern in the letter.
Vault.com could not be reached for comment.
Correction March 29, 2017
An earlier web version of this story mistakenly indicated that the case that spurred the Illinois attorney general to action involved Monster.com. The case involved ItsMyCareer.com, which dropped its resume-building feature.


Phaedra Parks To Represent Kandi Burruss In Lawsuit Against Kim Zolciak!
We previously reported that Real Housewives of Atlanta Star Kandi Burruss is suing her former friend Kim Zolciak over unpaid royalties and production fees for Kim's song, 'Don't Be Tardy.' RadarOnline is exclusively reporting that Kandi's BFF and RHOA co-star, Phaedra Parks will be representing her in the courtroom! According to the court documents: "Without Plaintiffs' authorization, license or consent, Plaintiffs' Sound Recording embodying a performance of Plaintiffs' Composition was commercially released as a single record… on or around, September 29, 2009 by Defendant TuneCore in agreement with Defendant Zolciak.
The plaintiff's also are seeking punitive damages, attorneys fees, and a jury trial.
Are YOU surprised Phaedra is representing Kandi in the lawsuit?
NEXT POST Next post: Video! Lisa Vanderpump Practices For First DWTS Live Show!


Picturesque river views from your spacious home in desirable Snug Harbor. Steps to the river and beach. Unique home with ceramic tile throughout entire home-No Carpet. Vaulted ceilings in kitchen and family. This home has lots of space and bonus rooms. Second master bedroom has it's own living area with kitchenette and separate entrance. Perfect for guest quarters or mother in laws. Fenced yard. Laundry hookup inside home. Launch your boat 5 blocks away via Ramp Road. Bus line and free beachside parking at 13th street.
I was searching for a Property and found this listing (MLS® #841291). Please send me more information regarding 1132 S Brevard Avenue, Cocoa Beach, FL, 32931. Thank you!
I'd like to request a showing of 1132 S Brevard Avenue, Cocoa Beach, FL, 32931 (MLS® #841291). Thank you!


12 acres of elevated wooded land with views to both the Caribbean Sea and the Atlantic Ocean.
Photo of the view towards the Caribbean Sea.
The land is located in the slands interior and is about 1 hours drive from Roseau and 35 mins drive from Melville Hall Airport.
Photo of the view towards the nearest neighbouring properties.
Photo showing the access track to the land.
The land bounds with and rises above the access track.
The land is presently covered with small trees.
Photo of the view towards the Atlantic Ocean.
The land would make a perfect location for a small eco-resort or large residential property.


text:
2019 TRADE DEADLINE: PROSPECT RECAP
Our new 2019 pro-side video, scouting report, and spotlight libraries are now live! Check out the links below–and you can always refer to our 2018 libraries for even more player info:
Pro Video Library (2019, 2018)
Pro Scouting Report Library (2019, 2018)
Pro Spotlight Library (2019, 2018)
FV 55+ PROSPECTS
Taylor Trammell, OF
From: Reds | To: Padres | Date: July 31, 2019
Ceiling: 55 Risk: High ETA: 2021
Ht/Wt: 6'2" / 194 lbs. B/T: L / L Highest Level: AA Age (as of July 31, 2019): 21y, 10m
Video #1 | Video #2 | Scouting Report
Trammell placed in the top-20 of our most recent Top 125 list entering 2019, though the young outfielder has scuffled in his first taste of Double-A. He's an excellent athlete who receives high marks for makeup and aptitude, which paired with his age and level allows reason to believe Trammell can make long term adjustments. Originally a CF, he has moved nearly full-time to a corner this year and will likely be limited to LF given a below-average throwing arm. Even so, Trammell's hitting ability, on-base skill, and plus-plus wheels still give the ceiling of a FV 55 prospect whose sum-of-parts can make him better than a regular in the best-case scenario. Though it hasn't fully translated to game action, this will be especially true if he can harness his above-average raw power and become more of a home run threat at maturity. Though he'll likely slide just a bit when we re-rank the mid-season Top 125, it's worth remembering Trammell is the same age as most college juniors in this year's draft class. He's a prominent prospect acquisition by San Diego and should contribute at the big league level for a long time.
Jazz Chisholm, SS
From: Diamondbacks | To: Marlins | Date: July 31, 2019
Ht/Wt: 5'11" / 165 lbs. B/T: L / R Highest Level: AA Age (as of July 31, 2019): 21y, 5m
Chisholm's enticing blend of youth, center-diamond athleticism, and power potential placed him among baseball's best infield prospects heading into 2019. He cemented his status as a top prospect with an excellent Fall League performance, though the 21-year-old struggled with an aggressive assignment to Double-A this year. His tools have still showed through in 2019 despite holes in his stat line, swatting 18 home runs and swiping 13 bases at the time of the trade. Chisholm is too aggressive at the plate, evidenced by a strikeout rate above 30-percent right now against older competition. If he can cut down on the whiffs and improve his hit tool, the ceiling is (at least) an impactful FV 55 shortstop with tools on both sides of the ball. Even if he doesn't, Chisholm is enough of a defender at shortstop to still be an everyday player with some sneaky pop at a premium position. Though they gave up a valuable long term piece (Zac Gallen) in order to acquire him, the Marlins are returning a very high-ceiling piece.
Jesus Sanchez, OF
From: Rays | To: Marlins | Date: July 31, 2019
Ht/Wt: 6'3" / 230 lbs. B/T: L / R Highest Level: AAA Age (as of July 31, 2019): 21y, 9m
Video | Scouting Report
Sanchez was the lone prospect in a trade between Tampa Bay and Miami, heading to the Marlins (along with big league reliever Ryne Stanek) in exchange for Major League pieces Trevor Richards and Nick Anderson. He has prototype corner outfield tools, with the chance to produce average and power and impact the game defensively in RF. Sanchez has performed while being young for the level each of the last two years, recently getting a promotion to Triple-A before the trade after slashing .275/.332/.404 in the Southern League. His strikeout rates and overall bat control have been encouraging for a young power hitter, though his plus raw power still hasn't fully come into games. There's reason to believe it will, though, as the 21-year-old is still filling out a large, long-levered frame and doesn't lift the ball much. Especially if he grows into more power, Sanchez checks the boxes of an above-average regular and could be a familiar face in the big leagues for years to come.
Corbin Martin, RHP
From: Astros | To: Diamondbacks | Date: July 31, 2019
Ht/Wt: 6'2" / 200 lbs. B/T: R / R Highest Level: MLB Age (as of July 31, 2019): 23y, 7m
We were fairly early on the Corbin Martin train and marked him as a Top 125 dating back to the fall of 2018, but the righty has broke out even more since then. He was the latest in a long line of Astros development success stories, finishing his first full pro season in Double-A and opening 2019 with excellent results in the hitter-friendly Pacific Coast League. Martin showed glimpses of his upside in five big league starts, then was optioned back to Triple-A and blew out his elbow there. He will miss the rest of this season–and likely most of 2020–recovering from Tommy John surgery, though the tools are here to slide into a mid-rotation profile upon returning. Martin's mid-90s fastball averaged an impressive 95 mph in his Major League debut, and he's able to reach back for a few extra ticks past there. He favored a slider in the minors but now goes interchangeably with two breaking pitches, with both his slider and curveball grading solid-average or better. His changeup has developed into an effective offering that rounds out the arsenal.
FV 50 PROSPECTS
Logan Allen, LHP
From: Padres | To: Indians | Date: July 31, 2019
Ht/Wt: 6'3" / 200 lbs. B/T: L / L Highest Level: MLB Age (as of July 31, 2019): 22y, 2m
Video | Scouting Report | Spotlight
Allen only turned 22-years-old earlier this season but has already debuted in the big leagues. His polish allowed the former eighth-rounder to move quickly through the minors, splitting the last two years between the high minors and Major Leagues. Allen's fastball ranges anywhere from 90-to-95 mph, backed by a deep mix of solid secondaries and advanced pitchability. His Triple-A numbers are skewed given the massively hitter-friendly climate in the Pacific Coast League, though Allen has missed bats at a good clip. He's a nice mix of ceiling and safety, checking the boxes of a future reliable mid-rotation lefty.
Zac Gallen, RHP
From: Marlins | To: Diamondbacks | Date: July 31, 2019
Ht/Wt: 6'2" / 190 lbs. B/T: R / R Highest Level: MLB Age (as of July 31, 2019): 23y, 11m
We were higher on Gallen than most outlets entering 2019, even by placing him seventh overall on our Marlins Organizational Review. That speaks to the degree he has come on even stronger this season, dominating in Triple-A and more than holding his own through seven big league starts. Though he's basically a finished product, Gallen has proved capable of performing in the big leagues and can help a team right away–with loads of controllable seasons ahead of him to sweeten the deal. He will immediately slide into Arizona's rotation and has a chance to be a prominent member of the organization's pitching staff moving forward.
J.B. Bukauskas, RHP
Ht/Wt: 6'0" / 196 lbs. B/T: R / R Highest Level: AA Age (as of July 31, 2019): 22y, 9m
Bukauskas has always had impact stuff, though his shorter frame, mechanical effort, and iffy walk numbers have kept questions about a future move to the 'pen in play. He hasn't done much to dispel those, walking nearly 15-percent of opposing batters in Double-A this year despite posting huge strikeout numbers. Bukauskas' fastball touches the upper-90s early in starts, often tailing off as his pitch count rises. The slider is a true plus pitch and could be even more of a weapon facing lineups once, though there's enough of a changeup to dream on him remaining in the rotation. It's not completely clear which direction Bukauskas will break, profiling as either a #4 starter with more stuff than feel or a potential high-leverage weapon with closer potential.
Seth Beer, 1B/OF
Ht/Wt: 6'3" / 195 lbs. B/T: L / R Highest Level: AA Age (as of July 31, 2019): 22y, 10m
Beer was the third of three FV 50+ prospects Arizona received in the Zach Grienke blockbuster with Houston. He's a corner-only defender that might wind up landlocked at 1B, but it's all about the bat with Beer. He has a long track record of offensive performance, stemming back to a successful three-year career at Clemson that included the best offensive season by a power conference freshman in recent memory. Beer has the hit tool, power production, and on-base skill to potentially clear the high offensive bar at a bat-only position, coming with the upside of a valuable everyday player on a corner. There's risk because all his value is tied to offensive output; Beer also shows signs of carrying a bit of a platoon split against same-side pitching. His overall offensive game is fairly polished and could get him to the big leagues in the next two years.
Joey Wentz, LHP
From: Braves | To: Tigers | Date: 7/31/2019
Ht/Wt: 6'5" / 210 lbs. B/T: L / L Highest Level: AA Age (as of July 31, 2019): 21y, 9m
Wentz was the main prospect returned to the Tigers for big league closer Shane Greene. He reached Double-A for the first time . His 6-foot-5 and 210-pound frame is built to eat innings, coming with the upside of a durable #4 starter. Wentz isn't overpowering in terms of velocity, but his fastball is firm enough in the low-90s with feel for both a curveball and changeup. He hasn't managed contact well early in his pro career, allowing lots of aerial contact and working with more control than command. Wentz repeats his delivery well, but there are some moving parts he could live to smoothen out in order to work down in the zone more consistently.
Simeon Woods Richardson, RHP
From: Mets | To: Blue Jays | Date: July 28, 2019
Ceiling: 50 Risk: Extreme ETA: 2022
Ht/Wt: 6'3" / 210 lbs. B/T: R / R Highest Level: A+ Age (as of July 31, 2019): 18y, 10m
Woods Richardson was the Mets' second round pick from a Texas high school in 2018. Extremely young for the class, he's still 18-years-old and won't turn 19 until after his first full professional season. He was turning heads in the South Atlantic League as the circuit's youngest regular starter prior to the trade, posting a 2.51 ERA in six starts after the all-star break. The Mets had just promoted him to their High-A affiliate and Toronto has opted to keep him there, as the young righty was assigned to Dunedin's roster in the Florida State League. Woods Richardson has the ingredients of a solid mid-rotation starter, with a lively mid-90s fastball and two big league caliber off-speed pitches on a strong, projectable frame. There's risk with any teenage arm, but we're very enthusiastic about the upside here. Woods Richardson could take another step forward and ultimately move into the FV 55 tier when all is said and done.
Kolby Allard, LHP
From: Braves | To: Rangers | Date: July 30, 2019
Ht/Wt: 6'1" / 190 lbs. B/T: L / L Highest Level: MLB Age (as of July 31, 2019): 21y, 11m
Allard has been a notable prospect name for years, dating all the way back to high school. He was one of the top prep arms in the 2015 Draft, considered a potential top-10 selection until a back issue slid him to the middle of the first round. Allard's brief big league stint did not go well in 2018, though he has rebounded fairly well in Triple-A this season. His stuff has backed up the last few years, now operating with a fastball that doesn't get above the low-90s. Allard won't be the #2/#3 starter some hoped coming out of the draft, but there's still room to grow into a serviceable backend starter. If his stuff just isn't enough to consistently work back through lineups, Allard is at least a safe bet to soak up innings in a swingman/spot-starting role (FV 40).
Anthony Kay, LHP
Ht/Wt: 6'0" / 218 lbs. B/T: L / L Highest Level: AAA Age (as of July 31, 2019): 24y, 4m
Kay was the 31st overall pick in 2016, though his heavy college workload at the University of Connecticut caught up with him. The lefty didn't pitch in an official game until two years after signing due to injuries, and he was shaking off the rust for most of 2018. Kay has rebounded this year, breezing through Double-A and earning a promotion to Triple-A a few weeks before the trade. The hitter-friendly conditions of the minors' highest level are mostly to blame for his unsightly 6.61 ERA through six International League starts, though Kay's struggles keeping the ball on the ground could become more of a problem against better hitters. As a starter, his fastball sits in the low-90s with a solid curveball but average overall stuff. It's notable that Kay flashed mid-90s velocity in his Futures Game appearance this year, though he likely won't bring that type of arm-strength to a traditional rotation role. Already 24, he's close to ready and could compete for backend innings with the Blue Jays next year.
Lewin Diaz, 1B
From: Twins | To: Marlins | Date: August 27, 2019
Diaz' size and left-handed power potential kept him on scouts' radar moving through the lowest rungs of Minnesota's system, though he never put everything together at the plate in full-season ball prior to this season. He came into 2019 both fully healthy and in the best physical shape of his career, vaulting back into prospect territory with a resurgent offensive showing across two levels. Diaz torched the Florida State League–slashing a ridiculous .317/.351/.702 in May–before continuing the torrid pace after a promotion to Double-A. His long levers and loose, fast stroke through the zone generate huge raw power that could finish a 70-grade tool. There's a high offensive bar to clear to fit as an everyday player as a 1B-only prospect, but Diaz could mash enough to do just that. The 22-year-old posts enthusing strikeout numbers for a slugger but rarely walks, also struggling in tough same-side matchups at times. The rebuilding Marlins did well in this deal relative to the cost, acquiring an emerging power prospect with the potential to fill a role long term.
Mauricio Dubon, SS
From: Brewers | To: Giants | Date: July 31, 2019
Dubon missed most of last season with a torn ACL, though he made his big league debut this year with Milwaukee and has had a resurgent year at the plate in Triple-A. His massive spike in power is likely aided by the Triple-A hitting environment, but Dubon's bat-to-ball skill and contact profile both hint he could hit enough for regular playing time. The 25-year-old is a versatile defender that plays a big league shortstop and can move around the infield, attributes that raise his floor. The Giants have a track record of identifying and getting value out of this type of prospect.
Jameson Hannah, OF
From: Athletics | To: Reds | Date: July 31, 2019
Ht/Wt: 5'9" / 185 lbs. B/T: L / L Highest Level: A+ Age (as of July 31, 2019): 21y, 11m
Hannah's best trait is his contact ability, projecting as an average (to slightly above-average) hitter at the big league level with doubles power. Defensively, Hannah covers plenty of ground in the outfield and shows closing speed running balls down to both sides. While he has split most of his innings between CF and RF this year, a fringy to below-average arm likely limits him to CF or LF in the big leagues. He's more of an average runner than plus, able to steal the occasional bag on the strength of his instincts as opposed to burning speed. Despite his solid feel for the barrel, he's a bit of a tweener given an average glove in CF without big power. The likely outcome is a quality fourth outfielder or capable low-end regular.
Tristan Beck, RHP
From: Braves | To: Giants | Date: July 31, 2019
Ht/Wt: 6'4" / 165 lbs. B/T: R / R Highest Level: A+ Age (as of July 31, 2019): 23y, 1m
Beck was known more for his polish and pitchability than stuff at Stanford, but he's shown a bit more across the board in pro ball. His fastball sits in the low-90s but has topped out at 95-96 mph this year, and a sharp curveball is now thrown harder as well. Beck's upper-80s changeup flashes decent fade but is inconsistent. He could be a backend starter with an improved third pitch, though he'll likely move to the 'pen to focus on the fastball and curve if that can't happen. Durability is also a concern given Beck's narrow frame and past injury history.
Cody Ponce, RHP
From: Brewers | To: Pirates | Date: July 29, 2019
Ponce was the Brewers' second round pick in 2015 from the college ranks. Never short on size or velocity, his stuff played down in a rotation role. 2019 has been his first full season pitching exclusively in relief and the results have been enthusing. Ponce has posted much better strikeout numbers from the 'pen, also cutting his walk rate and inducing more ground ball contact. His fastball touches 95-96 mph in short stints, backed up by a solid cutter and sharp three-quarters breaking ball that both play better facing lineups once. Pittsburgh assigned the 25-year-old to their Double-A affiliate, though he's a fairly finished product that might crack the big league roster late this year–and certainly by 2020–barring injury or something unexpected. Ponce projects as a standard bullpen piece with the upside of a solid 7th inning type.
Joshua Rojas, INF/OF
Ht/Wt: 6'1" / 185 lbs. B/T: R / R Highest Level: AAA Age (as of July 31, 2019): 25y, 1m
Rojas is the type of hard-hitting, multi-positional player the Astros have utilized wisely in years past. The fourth piece in Houston's return to the Diamondbacks for Zach Greinke, Rojas was an unheralded late-round pick from the University of Hawaii in 2017. He has climbed the ladder quickly with strong offensive showings at each level, though at the age of 25, Rojas doesn't project much more past this point. He's a nice sleeper added by a prudent Diamondbacks' scouting group, one that could step in to a bat-first utility role in short order.
Thomas Hatch, RHP
From: Cubs | To: Blue Jays | Date: July 30, 2019
Ht/Wt: 6'1" / 200 lbs. B/T: R / R Highest Level: AA Age (as of July 31, 2019): 24y, 10m
Hatch's raw stuff seems to fit the profile of a starting pitcher, with a mid-90s fastball backed by a slider and changeup that both flash like effective off-speed pitches. Though he has both missed more bats and walked fewer hitters this year in Double-A, trouble managing contact has led to him getting hit hard at times. The absolute ceiling is a #5 starter, but Hatch more realistically comes with the upside of a controllable swingman/spot-starter capable of filling a number of smaller roles on a pitching staff.
Alex Lange, RHP
From: Cubs | To: Tigers | Date: July 31, 2019
Lange was the 30th pick in the draft just two years ago after a decorated college career at LSU. He has hit a wall in the time since, though, and seen his stuff back up significantly under the every-fifth-day grind of pro ball. Lange entered Chicago's system with a mid-90s fastball and power breaking ball, but he has now morphed into a low-90s starter whose best off-speed is a changeup. The 23-year-old has scuffled across two levels in 2019 and was a good candidate for a change of scenery, an opportunity he'll get with Detroit. Lange's best chance at big league value now looks like it will come in the 'pen, where there's a chance he can get back to his stuff of old airing it out in short stints.
Joe McCarthy, OF
From: Rays | To: Giants | Date: 7/31/2019
The Rays added McCarthy to their 40-Man Roster in advance of last year's Rule 5 Draft, though he has continued to struggle to stay on the field. Tampa Bay will clear a roster spot without needing to DFA him, opting to send McCarthy to the Giants in a small trade instead. The 25-year-old has continually battled injuries and 2019 has been no different. He was placed on the IL to begin the season and didn't get back into game action with Triple-A Durham until June. He has an excellent eye at the plate and draws a ton of walks, showing raw power in BP that has yet to carry over to a full season. McCarthy had some speed early in his career, but a thickening frame–paired with a few years' worth of bumps and busies–has taken its toll. The Giants hope they're scoring a potential bench or platoon bat on the cheap and hoping for a rebound.
Paul Richan, RHP
From: Cubs | To: Tigers | Date: 7/31/2019
Richan commands a low-90s fastball well, able to throw strikes with four pitches. His best off-speed is a low-80s slider, and he'll mix a workable changeup and curve as well. None of his offerings grade above-average, but his strikethrowing and pitchability could get him to the big leagues in a long relief/spot-starter role. Richan doesn't miss a ton of bats and struggles to manage contact, with ground ball rates under 40-percent in each of his two pro seasons. He was named an all-star in the Carolina League this season.
Ray Black, RHP
From: Giants | To: Brewers | Date: 7/31/2019
Black is one of the hardest throwers in pro baseball, consistently cracking triple-digits and averaging a ridiculous 99 mph in his big league outings this year. His power slider can also play as a miss-bat pitch, and the combination of 80-grade velo and a breaking ball has allowed Black to rack up huge strikeout numbers throughout his career. Injuries and walks limit his ceiling, and he's remarkably still prospect eligible at the age of 29. The power stuff could continue to afford him opportunities at the Major League level as a standard bullpen piece.
Taylor Guilbeau, LHP
From: Nationals | To: Mariners | Date: July 31, 2019
Guilbeau put himself on the prospect radar by transitioning to the bullpen and started to get some buzz during Fall League last year. He kept that up this season, albeit as a 26-year-old in Double-A for the first time pitching exclusively in relief. His value is driven by proximity and a high floor more than upside, as Guilbeau doesn't fit as more than a middle relief profile but could pitch in the big leagues quickly. His fastball touches the mid-90s from a deceptive delivery, backed by a hard slider in the 84-to-88 mph range.
Scott Moss, LHP
From: Reds | To: Indians | Date: July 31, 2019
Moss was Cincinnati's fourth-round pick from the University of Florida in 2016. Injuries and a deep Gators rotation clouded his draft stock and made it tough for him to get innings, but he's progressed steadily up the ladder in the time since. None of Moss' pitches grade above-average, though improved spin and arsenal depth have lead to a career-high strikeout rate in Double-A. Unfortunately, his walks have also ticked up, and the lefty has never managed contact particularly well. He's the type of swingman/spot-starter candidate who might sneak into the back of a lesser rotation but pitches below that for a contender.
Travis Demeritte, OF
From: Atlanta | To: Detroit | Date: July 31, 2019
Ht/Wt: 6'0" / 180 lbs. B/T: R / R Highest Level: AAA Age (as of July 31, 2019): 24y, 10m
Demeritte has basically moved full-time to the outfield, though he played numerous infield positions earlier in his career. He has always had big power, historically held back by a high-strikeout approach and poor hit tool. The 24-year-old has turned a corner offensively this year in Triple-A, though it's tough to fully buy in given the inflated hitting environment at the level right now. Demeritte never cracked .240 in the high minors prior to this season, though he slashed .286/.387/.558 with Gwinnett in 2019. If he can reach base at a respectable clip and continue to cut down on strikeouts, the power and potential for defensive versatility could make Demeritte a bench player. He'll wind up a 4A/callup type that spends more time in Triple-A than the big leagues if those changes don't take hold at the highest level.
Jaylin Davis, OF
From: Minnesota | To: San Francisco | Date: July 31, 2019
Davis has worked his way up the ladder, now on the cusp of the big leagues after entering pro ball as an unknown 24th round pick. He has always shown raw power, though the question historically has been how much of it he'll tap into in game action. Davis has started answering some of those questions, moving into the FV 40 tier on the strength of a big year at the plate, including a .331/.405/.708 line in 41 Triple-A games prior to the trade. The ceiling is a slugging bench or platoon bat, though Davis will quickly fall into 4A depth territory if he doesn't hit at the big league level.
Ismael Aquino, RHP
From: Athletics | To: Royals | Date: July 27, 2019
Ht/Wt: 6'2" / 170 lbs. B/T: R / R Highest Level: R Age (as of July 31, 2019): 20y, 10m
Aquino is a lottery ticket pitching prospect that has yet to pitch above the complex level. The 20-year-old righty touches the high-90s with his fastball and sits at 94-95 mph, flashing a quality split-like changeup in the high-80s as the man off-speed. His mid-80s slider/cutter hybrid is a fringy pitch, and considering his below-average control, Aquino is likely to move to relief down the road. There's plenty of risk here, but the mix of youth, physicality, and arm-strength bump him into the FV 40 tier.
Prelander Berroa, RHP
From: Twins | To: Giants | Date: 7/31/2019
Ht/Wt: 5'11" / 170 lbs. B/T: R / R Highest Level: R Age (as of July 31, 2019): 19y, 3m
A short, hard-throwing righty, Berroa's fastball reaches the mid-90s from a quick arm and smooth delivery. His 79-to-82 mph curve has some power to it, giving the upside of a solid future offering. Like most teenage arms in the lottery ticket category, his changeup and control/command are fairly raw. Berroa is a wildcard prospect, though he sneaks into the FV 40 tier due to more athleticism and feel for spin than most fastball-only pitching prospects below the full-season level.
Kyle Johnston, RHP
From: Nationals | To: Blue Jays | Date: July 31, 2019
Ceiling: 35+ Risk: Follow ETA: 2020
Johnston has worked mostly as a starter in pro ball but profiles better in relief long term. His low-90s fastball could play up in short stints, aided by its above-average movement and late ride. Neither of his off-speed pitches project to miss bats, and even with a few extra ticks of velocity, he's not working with a true plus offering. Johnston could move quickly in the 'pen and has a chance to get big league time in a middle relief role.
Dairon Blanco, OF
Blanco is 26-years-old, but the Cuban outfielder shouldn't be compared directly to other similarly-aged players because he only has been in pro ball since 2017. He's a speedy CF who can play all three positions in the grass and has shown a bit more life at the plate in Double-A this year. Blanco profiles as a solid upper-level depth piece with the speed and defensive versatility to get cups of coffee.
Cal Stevenson, OF
From: Blue Jays | To: Astros | Date: July 31, 2019
Ht/Wt: 5'10" / 175 lbs. B/T: L / L Highest Level: A+ Age (as of July 31, 2019): 22y, 10m
Stevenson's tools grade as a tweener, lacking the power for a corner or speed and defense for a premium position. He's having a nice year in the Florida State League and displaying encouraging zone control, walking nearly as much as he has struck out. Stevenson makes plenty of contact, though there isn't a ton behind it. He projects for minimal power and puts most of his balls in play on the ground, something that often causes issues at higher levels for players that aren't plus runners. He's the type of prospect that often flames out in the upper-minors but can occasionally simply perform their way into a small big league role.
Aaron Fletcher, LHP
Fletcher was an 18th round pick in 2018 from the University of Houston. He breezed through two A-Ball levels in his first full pro season, receiving a promotion to Double-A shortly before being dealt to Seattle. A funky lefty reliever, Fletcher relies on deception and command over what's mostly a two-pitch mix. His fastball sits in the low-90s and tops out around 94 mph, backed by a sweepy slider at 78-to-81 mph he keeps from the same slot. Fletcher has both control and command, able to throw consistent strikes down in the zone to both sides of the plate. He hides the ball well from a closed front side that gives hitters a late look. There isn't a ton of ceiling for this type of prospect, but it's a profile that often reaches the big leagues in a smaller capacity.
Raider Uceta, OF/1B
From: Astros | To: Angels | Date: July 31, 2019
Ht/Wt: 6'0" / 215 lbs. B/T: L / L Highest Level: R Age (as of July 31, 2019): 18y, 6m
Uceta is a stocky, bat-only prospect that likely moves to 1B full-time down the road. There's feel for the barrel and advanced contact skill for his age, with reason to project on more power given his youth and thick frame. Scouts are concerned about the body and know there will be a high offensive bar to clear as a young player already landlocked on a corner.
Victor Nova, INF
Ht/Wt: 5'9" / 160 lbs. B/T: L / R Highest Level: R Age (as of July 31, 2019): 19y, 6m
Nova is an interesting lowest-minors sleeper that served as a flier to pad Cleveland's return for Trevor Bauer. He's an offensive-minded utility player that shows intriguing hitting tools and feel for the barrel. Nova can play a number of positions (2B, 3B, even some outfield) but doesn't project as a center-diamond defender. He is a follow to keep an eye on for now, with the ceiling of a versatile bench player (FV 40) if he continues on an upward trajectory higher up the ladder.
Jeffry Abreu, RHP
From: Dodgers | To: Cardinals | Date: 7/31/2019
Ht/Wt: 6'4" / 200 lbs. B/T: R / R Highest Level: R Age (as of July 31, 2019): 19y, 6m
Abreu moved from the Dodgers to St. Louis in the small deal that sent Jed Gyorko to Los Angeles. He's a teenage righty who has yet to pitch above the complex level and was in the midst of his stateside debut before the trade. Abreu has good size and physical projection remaining, already able to scrape 93-94 mph at best with his fastball and sit in the low-90s.
Elvis Alvarado, RHP
Alvarado signed with the Nationals as an outfielder for $700K in 2016. He only appeared in official games as a hitter in one season, an ill-fated offensive campaign in the DSL back in 2017. Washington converted him to the mound last year, though he's more pieces than final product and too raw to place in the FV 40 tier or higher. Alvarado has a live, loose arm and shows above-average velocity on his fastball that still projects a bit. His off-speed, control, and feel to pitch are in the early stages.
Previous articleSCOUTING PREVIEW: TRIPLE-A ALL-STAR GAME
Next article2020 MLB Draft Mock Draft (2080 Prefs)


TVS Motor Company Limited is the second largest two-wheeler and three-wheeler manufacturer in India producing scooters, motorcycles, mopeds, and three-wheelers. It ranks amongst the top ten two-wheeler companies in the world, distributing to over 60 countries. "Joy of riding" defines TVS's design sensibilities. TVS Motor's strength lies in the design and development of new products.
Bengaluru by Design invited designers to redesign the TVS Apache RR 310, and the NTorq 125. The aim of this contest was to showcase the power of good design by rejuvenating the two-wheelers in their respective categories, whilst staying true to the brand persona.
We choose to redesign the TVS Apache RR310, and it is with excitement and gratitude that we announce that our design has bagged the third position in this competition.
The graphics here give an inherent sense of built-up energy, just waiting to burst out. The red and blue sweeping lines depict speed and precision, coupled with the rounded edges that show versatility. A hint of the yin and yang concept lies encrypted in the graphics, in the form of a small element of blue in close vicinity of the red, and vice versa with the red element in the blue.
The overall composition exudes a feeling of fierce yet refined, powerful yet polished quickness. Even when the vehicle is standing still – it'd look as if it's beckoning to you, to take it to the track where it can truly show its prowess.
During the course of this competition, we realized that graphic designing for vehicles is something that really excites us, and is a space we would really like to explore. Sticking true to our resolution of trying out new and exciting areas this year, we will work towards further exploring this space.