text:
\section*{Acknowledgments}
Authors thank
Patrick Lewis,
Aleksandra Piktus,
for helpful discussions and technical support.

\section{Introduction} \label{sec:intro}
Entity Linking~\citep[EL,][]{hoffart-etal-2011-robust,dredze-etal-2010-entity,bunescu-pasca-2006-using,cucerzan-2007-large} is an important task in NLP, with plenty of applications in multiple domains, spanning Question Answering~\citep{de-cao-etal-2019-question,nie-etal-2019-revealing,Asai2020Learning}, Dialogue~\citep{bordes2016learning,wen-etal-2017-network,williams-etal-2017-hybrid,chen-etal-2017-robust,curry2018alana}, Biomedical systems~\citep{f1852247075c44dc8a462b92b17a55f4,zheng2015entity}, to name just a few. It consists of grounding entity mentions in unstructured texts to KB descriptors (\textit{e.g.}, Wikipedia articles).

The multilingual version of the EL problem has been for a long time tight to a purely cross-lingual formulation~\citep[XEL,][]{mcnamee-etal-2011-cross,ji2015overview}, where mentions expressed in one language are linked to a KB expressed in another (typically English).
Recently,~\citet{botha-etal-2020-entity} made a step towards a more inherently multilingual formulation by defining a language-agnostic KB, obtained by grouping language-specific descriptors per entity. Such formulation has the power of considering entities that do not have an English descriptor (\textit{e.g.}, a Wikipedia article in English) but have one in some other languages.

\input{figures/mGENRE}

A common design choice to most current solutions, regardless of the specific formulation, is to provide a unified entity representation, either by collating multilingual descriptors in a single vector or by defining a canonical language. For the common bi-encoder approach ~\citep{wu-etal-2020-scalable, botha-etal-2020-entity}, this might be optimal. However, in the recently proposed \textsc{GENRE}\@\xspace model ~\citep{decao2021autoregressive}, an autoregressive formulation to the EL problem leading to stronger performance and considerably smaller memory footprints than bi-encoder approaches on monolingual benchmarks, the representations to match against are entity names (\textit{i.e.}, strings) and it's unclear how to extend those beyond a monolingual setting.

In this context, we find that maintaining as much language information as possible, hence providing multiple representations per entity, helps due to the connections between source language and entity names in different languages. We additionally find that using all available languages as targets and aggregating over the possible choices is an effective way to deal with a zero-shot setting where no training data is available for the source language.

Concretely, in this paper, we present m\textsc{GENRE}\@\xspace, the first multilingual EL system that exploits a sequence-to-sequence architecture to generate entity names in more than 100 languages left to right, token-by-token in an autoregressive fashion and conditioned on the context (see Figure~\ref{fig:mGENRE} for an outline of our system).
While prior works use a single representation for each entity, we maintain entity names for as many languages as possible, which allows exploiting language connections between source input and target name.
To summarize, this work makes the following contributions:

\begin{itemize}[topsep=2pt,itemsep=-4pt]
    \item Extend the catalog of entity names by considering all languages for each entry in the KB. %
    Storing the multilingual names index is feasible and cheap (\textit{i.e.}, 2.2GB for $\sim$89M names).

    \item Design a novel objective function that marginalizes over all languages to perform a prediction. This approach is particularly effective in dealing with languages not seen during fine-tuning ($\sim$50\% improvements).

    \item Establish new state-of-the-art performance for the Mewsli-9~\citep{botha-etal-2020-entity}, TR2016\textsuperscript{hard}~\citep{tsai-roth-2016-cross} and TAC-KBP2015~\citep{ji2015overview} MEL datasets.

    \item Present extensive analysis of modeling choices, including the usage of candidates from a mention table, frequency-bucketed evaluation, and performance on an heldout set including low-resource languages.

    \item Publicly release our best model, pre-trained as multilingual denoising auto-encoder using the %
    BART objective \cite{lewis2019bart,liu2020multilingual} on large-scale monolingual corpora in 125 languages and fine-tuned to generate entity names given $\sim$730M in-context Wikipedia hyperlinks in 105 languages.

\end{itemize}
\section{Background}
We first introduce Multilingual Entity Linking in Section~\ref{sec:mel} highlighting its difference with monolingual and cross-lingual linking.
We address the MEL problem with a sequence-to-sequence model that generates textual entity identifiers (\textit{i.e.}, entity names). Our formulation generalizes the \textsc{GENRE}\@\xspace model by~\citet{decao2021autoregressive} to a multilingual setting (m\textsc{GENRE}\@\xspace). Thus in Section~\ref{sec:genre} and~\ref{sec:genre_bs}, we discuss the \textsc{GENRE}\@\xspace model and how it ranks entities with Beam Search respectively.

\subsection{Task Definition} \label{sec:mel}

\textit{Multilingual Entity Linking}~\citep[MEL,][]{botha-etal-2020-entity} is the task of linking a given entity mention $m$ in a given context $c$ of language $l \in \mathcal{L}_C$ to the corresponding entity $e \in \mathcal{E}$ in a multilingual Knowledge Base (KB).
See Figure~\ref{fig:mGENRE} for an example: there are textual inputs with entity mentions (in bold) and we ask the model to predict the corresponding entities in the KB.
A language-agnostic KB includes at least the name (but could include also descriptions, aliases, etc.) of each entity in one or more languages but there is no assumption about the relationship between these languages $\mathcal{L}_{KB}$ and languages of the context $\mathcal{L}_C$. This is a generalization of both monolingual Entity Linking EL and cross-lingual EL~\citep[XEL,][]{mcnamee-etal-2011-cross,ji2015overview}. The latter considers contexts in different languages while mapping mentions to entities in a monolingual KB (\textit{e.g.}, English Wikipedia).

Additionally, we assume that each $e \in \mathcal{E}$ has a unique textual identifier in at least a language. Concretely, in this work, we use Wikidata~\citep{vrandevcic2014wikidata} as our KB. Each Wikidata item lists a set of Wikipedia pages in multiple languages linked to it and in any given language each page has a unique name (\textit{i.e.}, its title).

\subsection{Autoregressive generation} \label{sec:genre}
\textsc{GENRE}\@\xspace ranks each $e \in \mathcal{E}$ by computing a score with an autoregressive formulation: $\mathrm{score}_\theta(e|x) = p_\theta(y|x) = \prod_{i=1}^N p_\theta(y_i|y_{<i},x)$ where $y$ is the sequence of $N$ tokens in the identifier of $e$, $x$ the input (\textit{i.e.}, the context $c$ and mention $m$), and $\theta$ the parameters of the model. \textsc{GENRE}\@\xspace is based on fine-tuned BART architecture~\citep{lewis2019bart} and it is trained using a standard seq2seq objective, \textit{i.e.}, maximizing the output sequence likelihood with teacher forcing~\citep{sutskever2011generating,sutskever2014sequence} and regularized with dropout~\citep{JMLR:v15:srivastava14a} and label smoothing~\citep{szegedy2016rethinking}.

\subsection{Ranking with Constrained Beam Search} \label{sec:genre_bs}
At test time, it is prohibitively expensive to compute a score for every element in $\mathcal{E}$ and then sort them. Thus, \textsc{GENRE}\@\xspace exploits Beam Search~\citep[BS,][]{sutskever2014sequence}, an established approximate decoding strategy to navigate the search space efficiently. Instead of explicitly scoring all entities in $\mathcal{E}$, search for the top-$k$ entities in $\mathcal{E}$ using BS with $k$ beams. BS only considers one step ahead during decoding (\textit{i.e.}, it generates the next token conditioned on the previous ones). Thus, \textsc{GENRE}\@\xspace employs a prefix tree (trie) to enable constrained beam search and then generate only valid entity identifiers.

\section{Model}

\label{sec:training}
To extend GENRE to a multilingual setting we need to define what are the unique identifiers of all entities in a language-agnostic fashion. This is not trivial since we rely on text representations that are by their nature grounded in some language. Concretely, for each entity $e$, we have a set of identifiers $\mathcal{I}_e$ that consists of pairs $\langle l, n_e^l \rangle$ where $l \in \mathcal{L}_{KB}$ indicates a language and $n_e^l$ the name of the entity $e$ in the language $l$. We extracted these identifiers from our KB---each Wikidata item has a set of Wikipedia pages in multiple languages linked to it, and in any given language, each page has a unique name. We identified 3 strategies to employ these identifiers:
\begin{enumerate}[label=\roman*),topsep=2pt,itemsep=-4pt]
    \item define a \textit{canonical} textual identifier for each entity such that there is a 1-to-1 mapping between the two (\textit{i.e.}, for each entity, select a specific language for its name---see Section~\ref{sec:canonical});
    \item define a N-to-1 mapping between textual identifier and entities concatenating a language ID (\textit{e.g.}, a special token or the ISO 639-1 code\footnote{\url{https://www.iso.org/standard/22109.html}}) followed by its name in that language---alternatively concatenating its name first and then a language ID (see Section~\ref{sec:plain_generation});
    \item treat the selection of an identifier in a particular language as a latent variable (\textit{i.e.}, we let the model learn a conditional distribution of languages given the input and we marginalize over those---see Section~\ref{sec:marginalization}).
\end{enumerate}
All of these strategies define a different way we compute the underlining likelihood of our model. In Figure~\ref{fig:mGENRE} we show an outline of m\textsc{GENRE}\@\xspace. The following subsections will present detailed discussions of the above 3 strategies.

\subsection{Canonical entity representation} \label{sec:canonical}
Selecting a single textual identifier for each entity corresponds to choosing its name among all the available languages of that entity.
We employ the same data-driven selection heuristic as in~\citet{botha-etal-2020-entity}: for each entity $e$ we sort all its names $n_e^l$ for each language $l$ according to the number of mentions of $e$ in documents of language $l$. Then we take the name $n_e^l$ in the language $l$ that has the most mentions of $e$. In case of a tie, we select the language that has the most number of mentions across all entities (\textit{i.e.}, the language for which we have more training data).
Having a single identifier for each entity corresponds to having a 1-to-1 mapping between strings and entities. Thus, $\mathrm{score}_\theta(e|x) = p_\theta(n_e|x)$ where with $n_e$ we indicate the \textit{canonical} name for $e$. We train to maximize the scores for all our training data. A downside of this strategy is that most of the time, the model
cannot exploit the lexical overlap between the context and entity name since it has to translate it in the canonical one.

\subsection{Multilingual entity representation} \label{sec:plain_generation}
To accommodate the canonical representation issues, we can predict entity names in any language.
Concatenating a language ID $l$ and an entity name $n_e^l$ in different orders induces two alternative factorizations. We train maximizing the scores for all our training data: $\mathrm{score}_\theta(e|x) =$
\begin{equation}
\begin{cases}
p_\theta(l|x) \cdot p_\theta(n_e^l | x, l) & \text{for `lang+name'} \\
p_\theta(n_e^l| x) \cdot p_\theta(l|n_e^l,x) & \text{for `name+lang'} \\
\end{cases}
\end{equation}
The former corresponds to first predicting a distribution over languages and then predicting a title \textit{conditioning} on the language $l$ where the latter corresponds to the opposite.
Predicting the language first conditions the generation to a smaller set earlier during beam search (\textit{i.e.}, all names in a specific language). However, it might exclude some targets from the search too early if the beam size is too small.
Predicting the language last does not condition the generation of names in a particular language but it \textit{asks} the model to disambiguate the language of the generated name whenever it is ambiguous (\textit{i.e.}, when the same name in different languages corresponds to possibly different entities). Only 1.65\% of the entity names need to be disambiguated with the language.
In practice, we observe no difference in performance between the two approaches.
Both strategies define an N-to-1 mapping between textual identifiers and entities and then at test time we just use a lookup table to select the correct KB item. This N-to-1 mapping is an advantage compared to using canonical names because the model can predict in any available language and therefore exploit synergies between source and target language as well as avoiding translation.

\subsection{Marginalization} \label{sec:marginalization}
Differently from the plain generation strategies described above, we can treat the textual identifiers as a latent variable and express $\mathrm{score}_\theta(e|x)$ as the probability of the entity name in all languages and marginalizing over them: $\mathrm{score}_\theta(e|x)=$
\begin{equation}
    p_\theta(e|x) = \sum_{\langle l, n_e^l \rangle \in \mathcal{I}_e} p_\theta(n_e^l,l|x) \;.
\end{equation}
Marginalization exposes the model to all representations in all languages of the same entity and it requires a minor modification of the training procedure. Unfortunately, because computing $\mathrm{score}_\theta(e|x)$ requires a sum over all languages, both training, and inference with marginalization are more expensive than with simple generation (scaling linearly with the number of languages).
However, at least during inference, we can still apply BS to only marginalize using the top-k generations.
For this reason, we test this training strategy only on few languages but we evaluate marginalization even when training with the other generation strategies described above.

\subsection{Candidate selection} \label{sec:candidates}
Modern EL systems that employ cross-encoding between context and entities usually do not score all entities in a KB as it is too computational expensive~\citep{wu-etal-2020-scalable}. Instead, they first apply candidate selection (with a less expensive method first or just a non-parametric mention table) to reduce the number of entities before scoring. In our formulation, there is no need to do that since m\textsc{GENRE}\@\xspace uses Beam Search to efficiently generate. However, using candidates might help, and therefore, we also experiment with that. Scoring all candidates might not be always possible (sometimes there are thousands of candidates for a mention) and especially when using an N-to-1 mapping between textual identifiers there will be names to rank in all languages available for each candidate. Thus, when we use candidates, it is to constrain BS steps further, rather than to rank all of them.

Concretely, candidate selection is made with an alias table. Using the training data, we build a mention table where we record all entities indexed by the names used to refer to them.
Additionally, we also use Wikipedia titles as additional mentions (useful for entities that never appear as links), redirects, Wikidata labels, and aliases.

\section{Experimental Setting}

We use Wikidata~\citep{vrandevcic2014wikidata} as our KB while exploiting the supervision signal from Wikipedia hyperlinks. For evaluation, we test our model on two established cross-lingual datasets, TR2016\textsuperscript{hard} and TAC-KBP2015~\citep{ji2015overview, tsai-roth-2016-cross}, as well as the recently proposed Mewsli-9 MEL dataset~\citep{botha-etal-2020-entity}.
Additionally, we propose a novel setting extracted from Wikinews\footnote{\url{https://www.wikinews.org}} where we train a model on a set of languages, and we test it on unseen ones.

\subsection{Knowledge Base: Wikidata}
We use Wikidata as the target KB to link to filtering the same heuristic as~\citet{botha-etal-2020-entity} (see Appedinx~\ref{app:experiments} for more details).
Eventually, our entity set $\mathcal{E}$ contains 20,277,987 items (as a reference, English Wikipedia has just $\sim$6M items). Using the corresponding Wikipedia titles as textual identifiers in all languages leads to a table of 53,849,351 entity names. We extended the identifiers including redirects which leads to a total of 89,270,463 entity names (see Table~\ref{tab:titles105} in Appendix~\ref{app:experiments} for more details).
Although large, the number of entity names is not a bottleneck as the generated prefix tree only occupies 2.2GB for storage. As a comparison the
\citepossessive{botha-etal-2020-entity} MEL systems need $\sim$10 times more storage.

\input{figures/stats105_langs}

\subsection{Supervision: Wikipedia}
For all experiments, we do not train a model from scratch, but we fine-tune a multilingual language model trained on 125 languages (see Appendix~\ref{app:experiments} for more details on the pre-trained model).
We exploit Wikipedia hyperlinks as the source of supervision for MEL. We used Wikipedia in 105 languages out of the $>$300 available. These 105 are all the languages for which our model was pre-trained on that overlaps with the one available in Wikipedia (see full language list in Figure~\ref{fig:stats105_langs} and more details in Appendix~\ref{app:experiments}).
Eventually, we extracted a large-scale dataset of 734,826,537 datapoints.
For the plain generation strategy, we selected as the ground truth the name in the source language. When such entity name is not available we randomly select 5 alternative languages and we use all of them as datapoints.
To enable model selection, we randomly selected 1k examples from each language to keep as a validation set.

\subsection{Datasets
} \label{sec:data_test}

\paragraph{Mewsli-9} \citep{botha-etal-2020-entity} contains 289,087 entity mentions appearing in 58,717 originally written news articles from
Wikinews, linked to WikiData. The corpus includes documents in 9 languages.\footnote{Arabic, English, Farsi, German, Japanese, Serbian, Spanish, Tamil, and Turkish.} Differently from the cross-lingual setting, this is a truly multilingual dataset since 11\% target entities in Mewsli-9 do not have an English Wikipedia page thus a XEL model would not link these.

\paragraph{TR2016\textsuperscript{hard}} \citep{tsai-roth-2016-cross} is a Wikipedia based cross-lingual dataset specifically constructed to contain difficult mention-entity pairs. Authors extracted Wikipedia hyperlinks for which the corresponding entity is not the most likely when using an alias table. Since we train on Wikipedia, to avoid an overlap with this test data, we removed all mentions from our training data that also appear in TR2016\textsuperscript{hard}. Note that this pruning strategy is more aggressive than~\citepossessive{tsai-roth-2016-cross} and~\citepossessive{botha-etal-2020-entity} strategies.
\citet{tsai-roth-2016-cross} assured to not have mention-entity pairs overlaps between training and test, but a mention (with a different entity) might appear in training.
\citet{botha-etal-2020-entity}\footnote{Information provided by private correspondence with the authors.} split at the page-level only, making sure to hold out all \citet{tsai-roth-2016-cross} test pages (and their corresponding pages in other languages), but they trained on any mention-entity pair that could be extracted from their remaining training page partition (\textit{i.e.}, they have overlap between training and text entity-mention pairs).
To compare with previous works~\citep{tsai-roth-2016-cross, upadhyay-etal-2018-joint, botha-etal-2020-entity} we only evaluate on German, Spanish, French and Italian (a total of 16,357 datapoints).

\paragraph{TAC-KBP2015} To evaluate our system on documents out of the Wikipedia domain, we experiment on the TAC-KBP2015 Tri-Lingual Entity Linking Track~\citep{ji2015overview}. To compare with previous works~\citep{tsai-roth-2016-cross, upadhyay-etal-2018-joint, sil2018neural, zhou-etal-2019-towards}, we use only Spanish and Chinese (\textit{i.e.}, we do not evaluate in English). Following previous work, we only evaluate \textit{in-KB} links~\citep{yamada-etal-2016-joint,ganea-hofmann-2017-deep}, i.e, we do not evaluate on mentions that link to entities out of the KB. Previous works considered Freebase~\citep{bollacker2008freebase} as KB, and thus we computed a mapping between Freebase ID and Wikidata ID. When we cannot solve the match, our system gets zero scores (\textit{i.e.}, it counts as a wrong prediction). TAC-KBP2015 contains 166 Chinese documents (84 news and 82 discussion forum articles) and 167 Spanish documents (84 news and 83 discussion forum articles) for a total of 12,853 mention-entity datapoints.

\paragraph{Wikinews-7} For the purpose of testing a model on languages unseen during training, we extract mention-entities pairs from Wikinews in 7 languages that are not in the Mewsli-9 language set.\footnote{Chinese, Czech, French, Italian, Polish, Portuguese, and Russian.} Table~\ref{tab:unseen_stats} in Appendix~\ref{app:experiments} reports statistics of this dataset. Wikinews-7 is created in the same way as Mewsli-9, but we used our own implementation to extract data from raw dumps\footnote{\citet{botha-etal-2020-entity} did not release code for extracting Mewsli-9 from a Wikinews dump.}.

\section{Results} \label{sec:results}

\input{tables/mewsli9_results}
\input{tables/tr2016_tackbp2015_results}

The main results of this work are reported in Table~\ref{tab:mewsli9} for Mewsli-9, and in Table~\ref{tab:tr2016_tackbp2015_results} for TR2016\textsuperscript{hard}, and TAC-KBP2015 respectively. Our m\textsc{GENRE}\@\xspace (trained with `title+lang') outperforms all previous works in all those datasets. We show the accuracy of m\textsc{GENRE}\@\xspace on the 105 languages in our Wikipedia validation set against an alias table baseline in Figure~\ref{fig:stats105_langs}. In Table~\ref{tab:stats105} in Appendix~\ref{app:experiments} we report more details on the validation set results.
In Table~\ref{tab:examples} we report some examples of correct and wrong predictions of our m\textsc{GENRE}\@\xspace on selected datapoints from Mewsli-9.

\subsection{Performance evaluation}

\paragraph{Mewsli-9}
In Table~\ref{tab:mewsli9} we compare our m\textsc{GENRE}\@\xspace against the best model from~\citet{botha-etal-2020-entity} (Model~F\textsuperscript{+}) as well as with their alias table baseline. We report results from m\textsc{GENRE}\@\xspace with and without constraining the beam search to the top-k candidates from the table (see Section~\ref{sec:candidates}) as well as with and without marginalization (see Section~\ref{sec:marginalization}). All of these alternatives outperform Model~F\textsuperscript{+} on both micro and macro average accuracy across the 9 languages. Our base model (without candidates or marginalization) has a 10.9\% error reduction in micro average and 18.0\% error reduction for macro average over all languages. The base model has no restrictions on candidates so it is effectively classifying among all the $\sim$20M entities. The base model performs better than Model~F\textsuperscript{+} on each individual language except English and German. Note that these languages are the ones for which we have more training data ($\sim$134M and $\sim$60M datapoints each) but also the languages that have the most entities/pages ($\sim$6.1M and $\sim$2.4M). Therefore these are the hardest languages to link.\footnote{Even though there are many datapoints, disambiguating between more entities is harder. Moreover, low-resource languages should be harder as there are fewer training instances, but there are no such languages in the Mewsli-9 set.} When enabling candidate filtering to restrict the space for generation, we further improve error reduction to 13.6\% and 21.0\% for micro and macro average respectively. Marginalization reduces the error by the same amount as candidate filtering but combining search with candidates and marginalization leads to our best model: it improves error reduction to 14.5\% and  23.0\% on micro and macro average respectively. Our best model is also better than Model~F\textsuperscript{+} in English and on par with it in German.

\paragraph{TR2016\textsuperscript{hard} and TAC-KBP2015}
We compared our m\textsc{GENRE}\@\xspace against cross-lingual systems~\citep{tsai-roth-2016-cross,sil2018neural,upadhyay-etal-2018-joint,zhou-etal-2019-towards} and Model~F\textsuperscript{+} by~\citet{botha-etal-2020-entity} in Table~\ref{tab:tr2016_tackbp2015_results}. Differently from Meswli-9, the base m\textsc{GENRE}\@\xspace model does not outperform previous systems. Using marginalization brings minimal improvements. Instead, using candidates gives +11\% absolute accuracy on TAC-KBP2015 and +5\% on TR2016\textsuperscript{hard} effectively making m\textsc{GENRE}\@\xspace state-of-the-art in both datasets. The role of candidates is very evident on TAC-KBP2015 where there is not much of a difference for Spanish but a +22\% absolute accuracy for Chinese. TAC-KBP2015 comes with a training set and we used it to expand the candidate set. Additionally, we also included all simplified Chinese versions of the entity names because we used traditional Chinese in pre-training, and TAC-KBP2015 uses simplified Chinese. Many mentions in  TAC-KBP2015 were not observed in Wikipedia, so the performance gain mostly comes from this but including the simplified and alternative Chinese names also played an important role (+5\% comes from this alone).\footnote{We speculate that including different version (\textit{e.g.}, different dialects for Arabic) of entity names could improve performance in all languages. Since this is not in the scope of this paper, we will leave it for future work.}

\input{tables/mewsli9_bins}

\subsection{Analysis}

\paragraph{By entity frequency}
Table~\ref{tab:mewsli_bins} shows a breakdown of Mewsli-9 accuracy by entity frequency in training for~\citepossessive{botha-etal-2020-entity} Model~F\textsuperscript{+} and m\textsc{GENRE}\@\xspace. Interestingly, our m\textsc{GENRE}\@\xspace has much higher accuracy (22\% vs 8\%) on unseen entities (\textit{i.e.}, the {[}0,1) bin). This is because our formulation can take advantage of copying names from the source, translating them or normalizing them. For example, an unseen person name should likely be linked to the entity with the same name. This is a powerful bias that gives the model advantage in these cases. On very rare entities (\textit{i.e.}, the {[}1,10) bin) our model performs worse than Model~F\textsuperscript{+}. Note that Model~F\textsuperscript{+} was trained specifically to tackle those cases (\textit{e.g.}, with hard negatives and frequency-based mini-batches) whereas our model was not. We argue that similar strategies can be applied to m\textsc{GENRE}\@\xspace to improve performance on rare entities, and we leave that to future work. The performance gap between Model~F\textsuperscript{+} and m\textsc{GENRE}\@\xspace on entities that appear more than 100 times in the training set is negligible.

\input{figures/stats_mentions}

\paragraph{By candidate frequency}
We additionally measure the accuracy on Mewsli-9 by the number of candidates retrieved from the alias table (details in Figure~\ref{fig:stats_mentions}). When there are no candidates ($\sim$12k datapoints that is $\sim$4\% Mewsli-9) an alias table would automatically fail, but m\textsc{GENRE}\@\xspace uses the entire KB as candidates and has 63.9\% accuracy. For datapoints with few candidates (\textit{e.g.}, less than 100), we could use m\textsc{GENRE}\@\xspace as a \textit{ranker} and score all of the options without relying on constrained beam search. However, this approach would be computationally infeasible when there are no candidates (\textit{i.e.}, we use all the KB as candidates) or too many candidates (\textit{e.g.}, thousands). Constrained BS allows us to efficiently explore the space of entity names, whatever the number of candidates.

\input{tables/wikinews33-ablation}
\input{figures/heatmaps_wikinews}

\paragraph{Unseen Languages}

We use our Wikinews-7 dataset to evaluate m\textsc{GENRE}\@\xspace capabilities to deal with languages not seen during training (\textit{i.e.}, the set of languages in train and test are disjoint).
This zero-shot setting implies that no mention table is available during inference; hence we do not consider candidates for test mentions.
We train our models on the nine Mewsli-9 languages and compare all strategies exposed in Section~\ref{sec:training}.
To make our ablation study feasible,
we restrict the training data to the first 1 million hyperlinks from Wikipedia abstracts.
Results are reported in Table~\ref{tab:unseen}.

\input{tables/examples}

Using our novel marginalization strategy that aggregates (both at training and inference time) over all seen languages to perform the linking brings an improvement of over 50\% with respect to considering a single language. To deeper investigate the behaviour of the model in this setting, we compute the probability mass distribution over languages seen at training time for the first prediction (reported in Figure~\ref{fig:heatmaps_wikinews}). When marginalization is enabled (Figure~\ref{fig:heatmaps_wikinews2}) the distribution is more spread across languages since the model is trained to use all of them. Hence the model can exploit connections between an unseen language and all seen languages for the linking process, which drastically increases the accuracy.

\section{Related Work}
The most related works to ours are~\citet{decao2021autoregressive}, that proposed to use an autoregressive language model for monolingual EL, and~\citet{botha-etal-2020-entity} that proposes to extend the cross-lingual EL task to multilingual EL with a language-agnostic KB.
We provide an outline of the GENRE model proposed by~\citet{decao2021autoregressive} in Section~\ref{sec:genre} and~\ref{sec:genre_bs}. GENRE was applied not only to EL but also for joint mention detection and entity linking (still with an autoregressive formulation) as well as to page-level document retrieval across multiple Knowledge Intensive Language Tasks~\citep[KILT;][]{petroni2020kilt} \textit{i.e.}, fact-checking, open-domain question answering, slot filling, and dialog.
\citepossessive{botha-etal-2020-entity} Model F\textsuperscript{+} is a \textit{bi-encoder} model: it is based on two BERT-based~\citep{devlin-etal-2019-bert} encoders that outputs vector representations for contet and entities. Similar to~\citet{wu-etal-2020-scalable} they rank entities with a dot-product between these representations. Model F\textsuperscript{+} uses the description of entities as input to the entity encoder and title, document and mention (separated with special tokens) as inputs to the context encoder.
Bi-encoders solutions may be memory inefficient since they require to keep in memory big matrices of embeddings, although memory-efficient dense retrieval has recently received attention ~\citep{izacard2020memory,min2021neurips,lewis2021paq}.

Another widely explored line of work is Cross-Language Entity Linking~\citep[XEL;][]{mcnamee2011cross, cheng-roth-2013-relational}. XEL considers contexts in different languages while mapping mentions to entities in a monolingual KB (\textit{e.g.}, English Wikipedia).
\citet{tsai2016cross} used alignments between languages to train multilingual entity embeddings. They used candidate selection and then they re-rank them with an SVM using these embeddings as well as a set of features (based on the multilingual title, mention, and context tokens). \citet{sil2018neural} explored the use of more sophisticated neural models for XEL as well as~\citet{upadhyay-etal-2018-joint} who jointly modeled type information to boost performance. \citet{zhou-etal-2019-towards} propose improvements to both entity candidate generation and disambiguation to make better use of the limited data in low-resource scenarios.
Note that in this work we focus on \textit{multilingual} EL, not cross-lingual. XEL is limiting to a monolingual KB (usually English), where MEL is more general since it can link to entities that might not be necessary represented in the target monolingual KB but in any of the available languages.

\section{Conclusion}
In this work, we propose an autoregressive formulation to the multilingual entity linking problem. For a mention in a given language, our solution generates entity names left-to-right and token-by-token.
The resulting system, m\textsc{GENRE}\@\xspace, maintains entity names in as many languages as possible to exploit language connections and interactions between source mention context and target entity name.
The constrained beam search decoding strategy enables fast search within a large set of entity names (\textit{e.g.}, the whole KB in multiple languages) with no need for large-scale dense indices.
We additionally design a novel objective function that marginalizes over all available languages to perform a prediction. We empirically show that this strategy is really effective in dealing with languages for which no training data is available (\textit{i.e.}, 50\%  improvements for languages never seen during training). Overall, our experiments show that m\textsc{GENRE}\@\xspace achieves new state-of-the-art performance on three popular multilingual entity linking datasets.

\section{Experimental Details} \label{app:experiments}

\subsection{Pre-training}
We used a pre-trained mBART~\citep{lewis2019bart,liu2020multilingual} model on 125 languages---see Figure~\ref{fig:venn_langs} for a visual overview of the overlap with these languages, Wikipedia and the languages used by~\citet{botha-etal-2020-entity}. mBART has 24 layers of hidden size is 1,024 and it has a total of 406M parameters.
We pre-trained on an extended version of the \texttt{cc100}~\citep{conneau-etal-2020-unsupervised,wenzek-etal-2020-ccnet} corpora available here\footnote{\url{http://data.statmt.org/cc-100}} where we increased the number of common crawl snapshots for low resource languages from 12 to 60.
The dataset has $\sim$5TB of text. We pre-trained for 500k steps with max 1,024 tokens per GPU on a variable batch size ($\sim$3000). Figure~\ref{fig:venn_langs} shows a Venn diagram on the overlap of languages used during pre-training and fine-tuning.

\subsection{Data for supervision}

\paragraph{Wikidata}
Wikidata contains tens of millions of items but most of them are scholarly articles or they correspond to help and template pages in Wikipedia (\textit{i.e.}, not entities we want to retain) \footnote{\url{https://www.wikidata.org/wiki/Wikidata:Statistics}}.
Following~\citep{botha-etal-2020-entity}, we only keep Wikidata items that have an associated Wikipedia page in at least one language, independent of the languages we actually model. Moreover, we filter out items that are a subclass (P279) or instance of
(P31) some Wikimedia organizational entities (\textit{e.g.}, help and template pages---see Table~\ref{tab:wikidata_filter}).

\paragraph{Wikipedia}
We aligned each Wikipedia hyperlink to its respective Wikidata item using a custom script. Note that each Wikipedia page maps to a Wikidata item. For the alignment we use i) direct reference when the hyperlink point directly to a Wikipedia page, ii) a re-directions table if the hyperlink points to an alias page, and iii) a Wikidata search among labels and aliases of items if the previous two alignment strategies failed. The previous two alignment strategies might fail when i) authors made a mistake linking on a non-existing page, ii) authors linked to a non-existing page on purpose hoping it will be created in the future, or iii) the original title of a page changed over time and no redirection was added to accommodate old hyperlinks. This procedure successfully aligns 91\% of the hyperlinks. We only keep unambiguous alignments since, when using Wikidata search (\textit{i.e.}, the third alignment strategy), the mapping could be ambiguous (\textit{e.g.}, multiple items may share the same labels and aliases).

In Table~\ref{tab:titles105} we report some statistics of the training data extracted from Wikipedia.
We use a standard Wikipedia extractor \texttt{wikiextractor}\footnote{\url{https://github.com/attardi/wikiextractor}} by~\citet{Wikiextractor2015} and a redirect extractor\footnote{\url{https://code.google.com/archive/p/wikipedia-redirect}}.
We use both Wikipedia and Wikidata dumps from 2019-10-01.

\input{figures/venn_langs}
\input{tables/wikidata_filter}

\subsection{Data for test}
We use Wikinews (from 2019-10-01) to construct our \textit{unseen} Wikinews-7 dataset. In Table~\ref{tab:unseen_stats} we report some statistic of our new dataset.

\input{tables/unseen_stats}

\subsection{Training}
We implemented, trained, and evaluate our model using the \texttt{fariseq} library~\citep{ott2019fairseq}. We trained mGENRE using Adam~\citep{kingma2014adam} with a learning rate $10^{-4}$, $\beta_1=0.9$, $\beta_2=0.98$, and with a linear warm-up for 5,000 steps followed by liner decay for maximum 2M steps. The objective is sequence-to-sequence categorical cross-entropy loss with 0.1 of label smoothing and 0.01 of weight decay. We used dropout probability of 0.1 and attention dropout of 0.1. We used max 3,072 tokens per GPU and variable batch size ($\sim$12,500). Training was done on 384 GPUs (Tesla V100 with 32GB of memory) and it completed in $\sim$72h for a total of $\sim$27,648 GPU hours or $\sim$1,152 GPU days. Since TAC-KBP2015 contains noisy text (\textit{e.g.}, XML/HTML tags), we further fine-tune mGENRE for 2k steps on its training set when testing on it.

\subsection{Inference}
At test time, we use Constrained Beam Search with 10 beams, length penalty of 1, and maximum decoding steps of 32. We restrict the input sequence to be at most 128 tokens cutting the left, right, or both parts of the context around a mention. When employing marginalization, we normalize the log-probabilities by sequence length using $\log p(y|x) / L^\alpha$, where $\alpha=0.5$ was tuned on the development set.

\section{Additional results} \label{app:additonal}

\subsection{Analysis}
\paragraph{By mention frequency}
We show a breakdown of the accuracy of m\textsc{GENRE}\@\xspace on Mewsli-9 by mention frequency in Table~\ref{tab:mewsli_bins_mentions}. The accuracy of unseen mentions is 66.7\% and increases up to 93.6\% for mentions seen more than 10k times. For extremely common mentions (\textit{i.e.}, seen more than 1M times) the accuracy drops to 73.2\%.
These mentions correspond to entities that are harder to disambiguate (\textit{e.g.}, `United States' appears 3.2M times but can be linked to the country as well as any sports team where the context refers to sports).

\paragraph{Unseen Languages}
Even though marginalization and canonical representation are the top-two systems in the unseen languages setting, they are not on seen languages. In Table~\ref{tab:ablation_mewsli} we report the results of all these strategies also on the seen languages (Mewsli-9 test set). Complementary to Figure~\ref{fig:heatmaps_wikinews} we also report the probability mass distribution over languages seen for Mewsli-9.

\input{tables/mewsli9_bins_mentions}
\input{tables/mewsli9_ablation}
\input{figures/heatmaps_mewsli}

\clearpage
\input{tables/titles105}

\clearpage
\input{figures/stats105_langs_large}
\input{tables/stats105_langs_large}

\clearpage
\input{figures/heatmap_all}

\section{Results} \label{sec:results}

\input{tables/mewsli9_results}
\input{tables/tr2016_tackbp2015_results}

The main results of this work are reported in Table~\ref{tab:mewsli9} for Mewsli-9, and in Table~\ref{tab:tr2016_tackbp2015_results} for TR2016\textsuperscript{hard}, and TAC-KBP2015 respectively. Our m\textsc{GENRE}\@\xspace (trained with `title+lang') outperforms all previous works in all those datasets. We show the accuracy of m\textsc{GENRE}\@\xspace on the 105 languages in our Wikipedia validation set against an alias table baseline in Figure~\ref{fig:stats105_langs}. In Table~\ref{tab:stats105} in Appendix~\ref{app:experiments} we report more details on the validation set results.
In Table~\ref{tab:examples} we report some examples of correct and wrong predictions of our m\textsc{GENRE}\@\xspace on selected datapoints from Mewsli-9.

\subsection{Performance evaluation}

\paragraph{Mewsli-9}
In Table~\ref{tab:mewsli9} we compare our m\textsc{GENRE}\@\xspace against the best model from~\citet{botha-etal-2020-entity} (Model~F\textsuperscript{+}) as well as with their alias table baseline. We report results from m\textsc{GENRE}\@\xspace with and without constraining the beam search to the top-k candidates from the table (see Section~\ref{sec:candidates}) as well as with and without marginalization (see Section~\ref{sec:marginalization}). All of these alternatives outperform Model~F\textsuperscript{+} on both micro and macro average accuracy across the 9 languages. Our base model (without candidates or marginalization) has a 10.9\% error reduction in micro average and 18.0\% error reduction for macro average over all languages. The base model has no restrictions on candidates so it is effectively classifying among all the $\sim$20M entities. The base model performs better than Model~F\textsuperscript{+} on each individual language except English and German. Note that these languages are the ones for which we have more training data ($\sim$134M and $\sim$60M datapoints each) but also the languages that have the most entities/pages ($\sim$6.1M and $\sim$2.4M). Therefore these are the hardest languages to link.\footnote{Even though there are many datapoints, disambiguating between more entities is harder. Moreover, low-resource languages should be harder as there are fewer training instances, but there are no such languages in the Mewsli-9 set.} When enabling candidate filtering to restrict the space for generation, we further improve error reduction to 13.6\% and 21.0\% for micro and macro average respectively. Marginalization reduces the error by the same amount as candidate filtering but combining search with candidates and marginalization leads to our best model: it improves error reduction to 14.5\% and  23.0\% on micro and macro average respectively. Our best model is also better than Model~F\textsuperscript{+} in English and on par with it in German.

\paragraph{TR2016\textsuperscript{hard} and TAC-KBP2015}
We compared our m\textsc{GENRE}\@\xspace against cross-lingual systems~\citep{tsai-roth-2016-cross,sil2018neural,upadhyay-etal-2018-joint,zhou-etal-2019-towards} and Model~F\textsuperscript{+} by~\citet{botha-etal-2020-entity} in Table~\ref{tab:tr2016_tackbp2015_results}. Differently from Meswli-9, the base m\textsc{GENRE}\@\xspace model does not outperform previous systems. Using marginalization brings minimal improvements. Instead, using candidates gives +11\% absolute accuracy on TAC-KBP2015 and +5\% on TR2016\textsuperscript{hard} effectively making m\textsc{GENRE}\@\xspace state-of-the-art in both datasets. The role of candidates is very evident on TAC-KBP2015 where there is not much of a difference for Spanish but a +22\% absolute accuracy for Chinese. TAC-KBP2015 comes with a training set and we used it to expand the candidate set. Additionally, we also included all simplified Chinese versions of the entity names because we used traditional Chinese in pre-training, and TAC-KBP2015 uses simplified Chinese. Many mentions in  TAC-KBP2015 were not observed in Wikipedia, so the performance gain mostly comes from this but including the simplified and alternative Chinese names also played an important role (+5\% comes from this alone).\footnote{We speculate that including different version (\textit{e.g.}, different dialects for Arabic) of entity names could improve performance in all languages. Since this is not in the scope of this paper, we will leave it for future work.}

\input{tables/mewsli9_bins}

\subsection{Analysis}

\paragraph{By entity frequency}
Table~\ref{tab:mewsli_bins} shows a breakdown of Mewsli-9 accuracy by entity frequency in training for~\citepossessive{botha-etal-2020-entity} Model~F\textsuperscript{+} and m\textsc{GENRE}\@\xspace. Interestingly, our m\textsc{GENRE}\@\xspace has much higher accuracy (22\% vs 8\%) on unseen entities (\textit{i.e.}, the {[}0,1) bin). This is because our formulation can take advantage of copying names from the source, translating them or normalizing them. For example, an unseen person name should likely be linked to the entity with the same name. This is a powerful bias that gives the model advantage in these cases. On very rare entities (\textit{i.e.}, the {[}1,10) bin) our model performs worse than Model~F\textsuperscript{+}. Note that Model~F\textsuperscript{+} was trained specifically to tackle those cases (\textit{e.g.}, with hard negatives and frequency-based mini-batches) whereas our model was not. We argue that similar strategies can be applied to m\textsc{GENRE}\@\xspace to improve performance on rare entities, and we leave that to future work. The performance gap between Model~F\textsuperscript{+} and m\textsc{GENRE}\@\xspace on entities that appear more than 100 times in the training set is negligible.

\input{figures/stats_mentions}

\paragraph{By candidate frequency}
We additionally measure the accuracy on Mewsli-9 by the number of candidates retrieved from the alias table (details in Figure~\ref{fig:stats_mentions}). When there are no candidates ($\sim$12k datapoints that is $\sim$4\% Mewsli-9) an alias table would automatically fail, but m\textsc{GENRE}\@\xspace uses the entire KB as candidates and has 63.9\% accuracy. For datapoints with few candidates (\textit{e.g.}, less than 100), we could use m\textsc{GENRE}\@\xspace as a \textit{ranker} and score all of the options without relying on constrained beam search. However, this approach would be computationally infeasible when there are no candidates (\textit{i.e.}, we use all the KB as candidates) or too many candidates (\textit{e.g.}, thousands). Constrained BS allows us to efficiently explore the space of entity names, whatever the number of candidates.

\input{tables/wikinews33-ablation}
\input{figures/heatmaps_wikinews}

\paragraph{Unseen Languages}

We use our Wikinews-7 dataset to evaluate m\textsc{GENRE}\@\xspace capabilities to deal with languages not seen during training (\textit{i.e.}, the set of languages in train and test are disjoint).
This zero-shot setting implies that no mention table is available during inference; hence we do not consider candidates for test mentions.
We train our models on the nine Mewsli-9 languages and compare all strategies exposed in Section~\ref{sec:training}.
To make our ablation study feasible,
we restrict the training data to the first 1 million hyperlinks from Wikipedia abstracts.
Results are reported in Table~\ref{tab:unseen}.

\input{tables/examples}

Using our novel marginalization strategy that aggregates (both at training and inference time) over all seen languages to perform the linking brings an improvement of over 50\% with respect to considering a single language. To deeper investigate the behaviour of the model in this setting, we compute the probability mass distribution over languages seen at training time for the first prediction (reported in Figure~\ref{fig:heatmaps_wikinews}). When marginalization is enabled (Figure~\ref{fig:heatmaps_wikinews2}) the distribution is more spread across languages since the model is trained to use all of them. Hence the model can exploit connections between an unseen language and all seen languages for the linking process, which drastically increases the accuracy.

\section{Introduction} \label{sec:intro}
Entity Linking~\citep[EL,][]{hoffart-etal-2011-robust,dredze-etal-2010-entity,bunescu-pasca-2006-using,cucerzan-2007-large} is an important task in NLP, with plenty of applications in multiple domains, spanning Question Answering~\citep{de-cao-etal-2019-question,nie-etal-2019-revealing,Asai2020Learning}, Dialogue~\citep{bordes2016learning,wen-etal-2017-network,williams-etal-2017-hybrid,chen-etal-2017-robust,curry2018alana}, Biomedical systems~\citep{f1852247075c44dc8a462b92b17a55f4,zheng2015entity}, to name just a few. It consists of grounding entity mentions in unstructured texts to KB descriptors (\textit{e.g.}, Wikipedia articles).

The multilingual version of the EL problem has been for a long time tight to a purely cross-lingual formulation~\citep[XEL,][]{mcnamee-etal-2011-cross,ji2015overview}, where mentions expressed in one language are linked to a KB expressed in another (typically English).
Recently,~\citet{botha-etal-2020-entity} made a step towards a more inherently multilingual formulation by defining a language-agnostic KB, obtained by grouping language-specific descriptors per entity. Such formulation has the power of considering entities that do not have an English descriptor (\textit{e.g.}, a Wikipedia article in English) but have one in some other languages.

\input{figures/mGENRE}

A common design choice to most current solutions, regardless of the specific formulation, is to provide a unified entity representation, either by collating multilingual descriptors in a single vector or by defining a canonical language. For the common bi-encoder approach ~\citep{wu-etal-2020-scalable, botha-etal-2020-entity}, this might be optimal. However, in the recently proposed \textsc{GENRE}\@\xspace model ~\citep{decao2021autoregressive}, an autoregressive formulation to the EL problem leading to stronger performance and considerably smaller memory footprints than bi-encoder approaches on monolingual benchmarks, the representations to match against are entity names (\textit{i.e.}, strings) and it's unclear how to extend those beyond a monolingual setting.

In this context, we find that maintaining as much language information as possible, hence providing multiple representations per entity, helps due to the connections between source language and entity names in different languages. We additionally find that using all available languages as targets and aggregating over the possible choices is an effective way to deal with a zero-shot setting where no training data is available for the source language.

Concretely, in this paper, we present m\textsc{GENRE}\@\xspace, the first multilingual EL system that exploits a sequence-to-sequence architecture to generate entity names in more than 100 languages left to right, token-by-token in an autoregressive fashion and conditioned on the context (see Figure~\ref{fig:mGENRE} for an outline of our system).
While prior works use a single representation for each entity, we maintain entity names for as many languages as possible, which allows exploiting language connections between source input and target name.
To summarize, this work makes the following contributions:

\begin{itemize}[topsep=2pt,itemsep=-4pt]
    \item Extend the catalog of entity names by considering all languages for each entry in the KB. %
    Storing the multilingual names index is feasible and cheap (\textit{i.e.}, 2.2GB for $\sim$89M names).

    \item Design a novel objective function that marginalizes over all languages to perform a prediction. This approach is particularly effective in dealing with languages not seen during fine-tuning ($\sim$50\% improvements).

    \item Establish new state-of-the-art performance for the Mewsli-9~\citep{botha-etal-2020-entity}, TR2016\textsuperscript{hard}~\citep{tsai-roth-2016-cross} and TAC-KBP2015~\citep{ji2015overview} MEL datasets.

    \item Present extensive analysis of modeling choices, including the usage of candidates from a mention table, frequency-bucketed evaluation, and performance on an heldout set including low-resource languages.

    \item Publicly release our best model, pre-trained as multilingual denoising auto-encoder using the %
    BART objective \cite{lewis2019bart,liu2020multilingual} on large-scale monolingual corpora in 125 languages and fine-tuned to generate entity names given $\sim$730M in-context Wikipedia hyperlinks in 105 languages.

\end{itemize}

\section{Experimental Setting}

We use Wikidata~\citep{vrandevcic2014wikidata} as our KB while exploiting the supervision signal from Wikipedia hyperlinks. For evaluation, we test our model on two established cross-lingual datasets, TR2016\textsuperscript{hard} and TAC-KBP2015~\citep{ji2015overview, tsai-roth-2016-cross}, as well as the recently proposed Mewsli-9 MEL dataset~\citep{botha-etal-2020-entity}.
Additionally, we propose a novel setting extracted from Wikinews\footnote{\url{https://www.wikinews.org}} where we train a model on a set of languages, and we test it on unseen ones.

\subsection{Knowledge Base: Wikidata}
We use Wikidata as the target KB to link to filtering the same heuristic as~\citet{botha-etal-2020-entity} (see Appedinx~\ref{app:experiments} for more details).
Eventually, our entity set $\mathcal{E}$ contains 20,277,987 items (as a reference, English Wikipedia has just $\sim$6M items). Using the corresponding Wikipedia titles as textual identifiers in all languages leads to a table of 53,849,351 entity names. We extended the identifiers including redirects which leads to a total of 89,270,463 entity names (see Table~\ref{tab:titles105} in Appendix~\ref{app:experiments} for more details).
Although large, the number of entity names is not a bottleneck as the generated prefix tree only occupies 2.2GB for storage. As a comparison the
\citepossessive{botha-etal-2020-entity} MEL systems need $\sim$10 times more storage.

\input{figures/stats105_langs}

\subsection{Supervision: Wikipedia}
For all experiments, we do not train a model from scratch, but we fine-tune a multilingual language model trained on 125 languages (see Appendix~\ref{app:experiments} for more details on the pre-trained model).
We exploit Wikipedia hyperlinks as the source of supervision for MEL. We used Wikipedia in 105 languages out of the $>$300 available. These 105 are all the languages for which our model was pre-trained on that overlaps with the one available in Wikipedia (see full language list in Figure~\ref{fig:stats105_langs} and more details in Appendix~\ref{app:experiments}).
Eventually, we extracted a large-scale dataset of 734,826,537 datapoints.
For the plain generation strategy, we selected as the ground truth the name in the source language. When such entity name is not available we randomly select 5 alternative languages and we use all of them as datapoints.
To enable model selection, we randomly selected 1k examples from each language to keep as a validation set.

\subsection{Datasets
} \label{sec:data_test}

\paragraph{Mewsli-9} \citep{botha-etal-2020-entity} contains 289,087 entity mentions appearing in 58,717 originally written news articles from
Wikinews, linked to WikiData. The corpus includes documents in 9 languages.\footnote{Arabic, English, Farsi, German, Japanese, Serbian, Spanish, Tamil, and Turkish.} Differently from the cross-lingual setting, this is a truly multilingual dataset since 11\% target entities in Mewsli-9 do not have an English Wikipedia page thus a XEL model would not link these.

\paragraph{TR2016\textsuperscript{hard}} \citep{tsai-roth-2016-cross} is a Wikipedia based cross-lingual dataset specifically constructed to contain difficult mention-entity pairs. Authors extracted Wikipedia hyperlinks for which the corresponding entity is not the most likely when using an alias table. Since we train on Wikipedia, to avoid an overlap with this test data, we removed all mentions from our training data that also appear in TR2016\textsuperscript{hard}. Note that this pruning strategy is more aggressive than~\citepossessive{tsai-roth-2016-cross} and~\citepossessive{botha-etal-2020-entity} strategies.
\citet{tsai-roth-2016-cross} assured to not have mention-entity pairs overlaps between training and test, but a mention (with a different entity) might appear in training.
\citet{botha-etal-2020-entity}\footnote{Information provided by private correspondence with the authors.} split at the page-level only, making sure to hold out all \citet{tsai-roth-2016-cross} test pages (and their corresponding pages in other languages), but they trained on any mention-entity pair that could be extracted from their remaining training page partition (\textit{i.e.}, they have overlap between training and text entity-mention pairs).
To compare with previous works~\citep{tsai-roth-2016-cross, upadhyay-etal-2018-joint, botha-etal-2020-entity} we only evaluate on German, Spanish, French and Italian (a total of 16,357 datapoints).

\paragraph{TAC-KBP2015} To evaluate our system on documents out of the Wikipedia domain, we experiment on the TAC-KBP2015 Tri-Lingual Entity Linking Track~\citep{ji2015overview}. To compare with previous works~\citep{tsai-roth-2016-cross, upadhyay-etal-2018-joint, sil2018neural, zhou-etal-2019-towards}, we use only Spanish and Chinese (\textit{i.e.}, we do not evaluate in English). Following previous work, we only evaluate \textit{in-KB} links~\citep{yamada-etal-2016-joint,ganea-hofmann-2017-deep}, i.e, we do not evaluate on mentions that link to entities out of the KB. Previous works considered Freebase~\citep{bollacker2008freebase} as KB, and thus we computed a mapping between Freebase ID and Wikidata ID. When we cannot solve the match, our system gets zero scores (\textit{i.e.}, it counts as a wrong prediction). TAC-KBP2015 contains 166 Chinese documents (84 news and 82 discussion forum articles) and 167 Spanish documents (84 news and 83 discussion forum articles) for a total of 12,853 mention-entity datapoints.

\paragraph{Wikinews-7} For the purpose of testing a model on languages unseen during training, we extract mention-entities pairs from Wikinews in 7 languages that are not in the Mewsli-9 language set.\footnote{Chinese, Czech, French, Italian, Polish, Portuguese, and Russian.} Table~\ref{tab:unseen_stats} in Appendix~\ref{app:experiments} reports statistics of this dataset. Wikinews-7 is created in the same way as Mewsli-9, but we used our own implementation to extract data from raw dumps\footnote{\citet{botha-etal-2020-entity} did not release code for extracting Mewsli-9 from a Wikinews dump.}.

\section*{Acknowledgments}
Authors thank
Patrick Lewis,
Aleksandra Piktus,
for helpful discussions and technical support.

\section{Background}
We first introduce Multilingual Entity Linking in Section~\ref{sec:mel} highlighting its difference with monolingual and cross-lingual linking.
We address the MEL problem with a sequence-to-sequence model that generates textual entity identifiers (\textit{i.e.}, entity names). Our formulation generalizes the \textsc{GENRE}\@\xspace model by~\citet{decao2021autoregressive} to a multilingual setting (m\textsc{GENRE}\@\xspace). Thus in Section~\ref{sec:genre} and~\ref{sec:genre_bs}, we discuss the \textsc{GENRE}\@\xspace model and how it ranks entities with Beam Search respectively.

\subsection{Task Definition} \label{sec:mel}

\textit{Multilingual Entity Linking}~\citep[MEL,][]{botha-etal-2020-entity} is the task of linking a given entity mention $m$ in a given context $c$ of language $l \in \mathcal{L}_C$ to the corresponding entity $e \in \mathcal{E}$ in a multilingual Knowledge Base (KB).
See Figure~\ref{fig:mGENRE} for an example: there are textual inputs with entity mentions (in bold) and we ask the model to predict the corresponding entities in the KB.
A language-agnostic KB includes at least the name (but could include also descriptions, aliases, etc.) of each entity in one or more languages but there is no assumption about the relationship between these languages $\mathcal{L}_{KB}$ and languages of the context $\mathcal{L}_C$. This is a generalization of both monolingual Entity Linking EL and cross-lingual EL~\citep[XEL,][]{mcnamee-etal-2011-cross,ji2015overview}. The latter considers contexts in different languages while mapping mentions to entities in a monolingual KB (\textit{e.g.}, English Wikipedia).

Additionally, we assume that each $e \in \mathcal{E}$ has a unique textual identifier in at least a language. Concretely, in this work, we use Wikidata~\citep{vrandevcic2014wikidata} as our KB. Each Wikidata item lists a set of Wikipedia pages in multiple languages linked to it and in any given language each page has a unique name (\textit{i.e.}, its title).

\subsection{Autoregressive generation} \label{sec:genre}
\textsc{GENRE}\@\xspace ranks each $e \in \mathcal{E}$ by computing a score with an autoregressive formulation: $\mathrm{score}_\theta(e|x) = p_\theta(y|x) = \prod_{i=1}^N p_\theta(y_i|y_{<i},x)$ where $y$ is the sequence of $N$ tokens in the identifier of $e$, $x$ the input (\textit{i.e.}, the context $c$ and mention $m$), and $\theta$ the parameters of the model. \textsc{GENRE}\@\xspace is based on fine-tuned BART architecture~\citep{lewis2019bart} and it is trained using a standard seq2seq objective, \textit{i.e.}, maximizing the output sequence likelihood with teacher forcing~\citep{sutskever2011generating,sutskever2014sequence} and regularized with dropout~\citep{JMLR:v15:srivastava14a} and label smoothing~\citep{szegedy2016rethinking}.

\subsection{Ranking with Constrained Beam Search} \label{sec:genre_bs}
At test time, it is prohibitively expensive to compute a score for every element in $\mathcal{E}$ and then sort them. Thus, \textsc{GENRE}\@\xspace exploits Beam Search~\citep[BS,][]{sutskever2014sequence}, an established approximate decoding strategy to navigate the search space efficiently. Instead of explicitly scoring all entities in $\mathcal{E}$, search for the top-$k$ entities in $\mathcal{E}$ using BS with $k$ beams. BS only considers one step ahead during decoding (\textit{i.e.}, it generates the next token conditioned on the previous ones). Thus, \textsc{GENRE}\@\xspace employs a prefix tree (trie) to enable constrained beam search and then generate only valid entity identifiers.

\section{Experimental Details} \label{app:experiments}

\subsection{Pre-training}
We used a pre-trained mBART~\citep{lewis2019bart,liu2020multilingual} model on 125 languages---see Figure~\ref{fig:venn_langs} for a visual overview of the overlap with these languages, Wikipedia and the languages used by~\citet{botha-etal-2020-entity}. mBART has 24 layers of hidden size is 1,024 and it has a total of 406M parameters.
We pre-trained on an extended version of the \texttt{cc100}~\citep{conneau-etal-2020-unsupervised,wenzek-etal-2020-ccnet} corpora available here\footnote{\url{http://data.statmt.org/cc-100}} where we increased the number of common crawl snapshots for low resource languages from 12 to 60.
The dataset has $\sim$5TB of text. We pre-trained for 500k steps with max 1,024 tokens per GPU on a variable batch size ($\sim$3000). Figure~\ref{fig:venn_langs} shows a Venn diagram on the overlap of languages used during pre-training and fine-tuning.

\subsection{Data for supervision}

\paragraph{Wikidata}
Wikidata contains tens of millions of items but most of them are scholarly articles or they correspond to help and template pages in Wikipedia (\textit{i.e.}, not entities we want to retain) \footnote{\url{https://www.wikidata.org/wiki/Wikidata:Statistics}}.
Following~\citep{botha-etal-2020-entity}, we only keep Wikidata items that have an associated Wikipedia page in at least one language, independent of the languages we actually model. Moreover, we filter out items that are a subclass (P279) or instance of
(P31) some Wikimedia organizational entities (\textit{e.g.}, help and template pages---see Table~\ref{tab:wikidata_filter}).

\paragraph{Wikipedia}
We aligned each Wikipedia hyperlink to its respective Wikidata item using a custom script. Note that each Wikipedia page maps to a Wikidata item. For the alignment we use i) direct reference when the hyperlink point directly to a Wikipedia page, ii) a re-directions table if the hyperlink points to an alias page, and iii) a Wikidata search among labels and aliases of items if the previous two alignment strategies failed. The previous two alignment strategies might fail when i) authors made a mistake linking on a non-existing page, ii) authors linked to a non-existing page on purpose hoping it will be created in the future, or iii) the original title of a page changed over time and no redirection was added to accommodate old hyperlinks. This procedure successfully aligns 91\% of the hyperlinks. We only keep unambiguous alignments since, when using Wikidata search (\textit{i.e.}, the third alignment strategy), the mapping could be ambiguous (\textit{e.g.}, multiple items may share the same labels and aliases).

In Table~\ref{tab:titles105} we report some statistics of the training data extracted from Wikipedia.
We use a standard Wikipedia extractor \texttt{wikiextractor}\footnote{\url{https://github.com/attardi/wikiextractor}} by~\citet{Wikiextractor2015} and a redirect extractor\footnote{\url{https://code.google.com/archive/p/wikipedia-redirect}}.
We use both Wikipedia and Wikidata dumps from 2019-10-01.

\input{figures/venn_langs}
\input{tables/wikidata_filter}

\subsection{Data for test}
We use Wikinews (from 2019-10-01) to construct our \textit{unseen} Wikinews-7 dataset. In Table~\ref{tab:unseen_stats} we report some statistic of our new dataset.

\input{tables/unseen_stats}

\subsection{Training}
We implemented, trained, and evaluate our model using the \texttt{fariseq} library~\citep{ott2019fairseq}. We trained mGENRE using Adam~\citep{kingma2014adam} with a learning rate $10^{-4}$, $\beta_1=0.9$, $\beta_2=0.98$, and with a linear warm-up for 5,000 steps followed by liner decay for maximum 2M steps. The objective is sequence-to-sequence categorical cross-entropy loss with 0.1 of label smoothing and 0.01 of weight decay. We used dropout probability of 0.1 and attention dropout of 0.1. We used max 3,072 tokens per GPU and variable batch size ($\sim$12,500). Training was done on 384 GPUs (Tesla V100 with 32GB of memory) and it completed in $\sim$72h for a total of $\sim$27,648 GPU hours or $\sim$1,152 GPU days. Since TAC-KBP2015 contains noisy text (\textit{e.g.}, XML/HTML tags), we further fine-tune mGENRE for 2k steps on its training set when testing on it.

\subsection{Inference}
At test time, we use Constrained Beam Search with 10 beams, length penalty of 1, and maximum decoding steps of 32. We restrict the input sequence to be at most 128 tokens cutting the left, right, or both parts of the context around a mention. When employing marginalization, we normalize the log-probabilities by sequence length using $\log p(y|x) / L^\alpha$, where $\alpha=0.5$ was tuned on the development set.

\section{Additional results} \label{app:additonal}

\subsection{Analysis}
\paragraph{By mention frequency}
We show a breakdown of the accuracy of m\textsc{GENRE}\@\xspace on Mewsli-9 by mention frequency in Table~\ref{tab:mewsli_bins_mentions}. The accuracy of unseen mentions is 66.7\% and increases up to 93.6\% for mentions seen more than 10k times. For extremely common mentions (\textit{i.e.}, seen more than 1M times) the accuracy drops to 73.2\%.
These mentions correspond to entities that are harder to disambiguate (\textit{e.g.}, `United States' appears 3.2M times but can be linked to the country as well as any sports team where the context refers to sports).

\paragraph{Unseen Languages}
Even though marginalization and canonical representation are the top-two systems in the unseen languages setting, they are not on seen languages. In Table~\ref{tab:ablation_mewsli} we report the results of all these strategies also on the seen languages (Mewsli-9 test set). Complementary to Figure~\ref{fig:heatmaps_wikinews} we also report the probability mass distribution over languages seen for Mewsli-9.

\input{tables/mewsli9_bins_mentions}
\input{tables/mewsli9_ablation}
\input{figures/heatmaps_mewsli}

\clearpage
\input{tables/titles105}

\clearpage
\input{figures/stats105_langs_large}
\input{tables/stats105_langs_large}

\clearpage
\input{figures/heatmap_all}

\section{Related Work}
The most related works to ours are~\citet{decao2021autoregressive}, that proposed to use an autoregressive language model for monolingual EL, and~\citet{botha-etal-2020-entity} that proposes to extend the cross-lingual EL task to multilingual EL with a language-agnostic KB.
We provide an outline of the GENRE model proposed by~\citet{decao2021autoregressive} in Section~\ref{sec:genre} and~\ref{sec:genre_bs}. GENRE was applied not only to EL but also for joint mention detection and entity linking (still with an autoregressive formulation) as well as to page-level document retrieval across multiple Knowledge Intensive Language Tasks~\citep[KILT;][]{petroni2020kilt} \textit{i.e.}, fact-checking, open-domain question answering, slot filling, and dialog.
\citepossessive{botha-etal-2020-entity} Model F\textsuperscript{+} is a \textit{bi-encoder} model: it is based on two BERT-based~\citep{devlin-etal-2019-bert} encoders that outputs vector representations for contet and entities. Similar to~\citet{wu-etal-2020-scalable} they rank entities with a dot-product between these representations. Model F\textsuperscript{+} uses the description of entities as input to the entity encoder and title, document and mention (separated with special tokens) as inputs to the context encoder.
Bi-encoders solutions may be memory inefficient since they require to keep in memory big matrices of embeddings, although memory-efficient dense retrieval has recently received attention ~\citep{izacard2020memory,min2021neurips,lewis2021paq}.

Another widely explored line of work is Cross-Language Entity Linking~\citep[XEL;][]{mcnamee2011cross, cheng-roth-2013-relational}. XEL considers contexts in different languages while mapping mentions to entities in a monolingual KB (\textit{e.g.}, English Wikipedia).
\citet{tsai2016cross} used alignments between languages to train multilingual entity embeddings. They used candidate selection and then they re-rank them with an SVM using these embeddings as well as a set of features (based on the multilingual title, mention, and context tokens). \citet{sil2018neural} explored the use of more sophisticated neural models for XEL as well as~\citet{upadhyay-etal-2018-joint} who jointly modeled type information to boost performance. \citet{zhou-etal-2019-towards} propose improvements to both entity candidate generation and disambiguation to make better use of the limited data in low-resource scenarios.
Note that in this work we focus on \textit{multilingual} EL, not cross-lingual. XEL is limiting to a monolingual KB (usually English), where MEL is more general since it can link to entities that might not be necessary represented in the target monolingual KB but in any of the available languages.

\section{Conclusion}
In this work, we propose an autoregressive formulation to the multilingual entity linking problem. For a mention in a given language, our solution generates entity names left-to-right and token-by-token.
The resulting system, m\textsc{GENRE}\@\xspace, maintains entity names in as many languages as possible to exploit language connections and interactions between source mention context and target entity name.
The constrained beam search decoding strategy enables fast search within a large set of entity names (\textit{e.g.}, the whole KB in multiple languages) with no need for large-scale dense indices.
We additionally design a novel objective function that marginalizes over all available languages to perform a prediction. We empirically show that this strategy is really effective in dealing with languages for which no training data is available (\textit{i.e.}, 50\%  improvements for languages never seen during training). Overall, our experiments show that m\textsc{GENRE}\@\xspace achieves new state-of-the-art performance on three popular multilingual entity linking datasets.

\section{Model}

\label{sec:training}
To extend GENRE to a multilingual setting we need to define what are the unique identifiers of all entities in a language-agnostic fashion. This is not trivial since we rely on text representations that are by their nature grounded in some language. Concretely, for each entity $e$, we have a set of identifiers $\mathcal{I}_e$ that consists of pairs $\langle l, n_e^l \rangle$ where $l \in \mathcal{L}_{KB}$ indicates a language and $n_e^l$ the name of the entity $e$ in the language $l$. We extracted these identifiers from our KB---each Wikidata item has a set of Wikipedia pages in multiple languages linked to it, and in any given language, each page has a unique name. We identified 3 strategies to employ these identifiers:
\begin{enumerate}[label=\roman*),topsep=2pt,itemsep=-4pt]
    \item define a \textit{canonical} textual identifier for each entity such that there is a 1-to-1 mapping between the two (\textit{i.e.}, for each entity, select a specific language for its name---see Section~\ref{sec:canonical});
    \item define a N-to-1 mapping between textual identifier and entities concatenating a language ID (\textit{e.g.}, a special token or the ISO 639-1 code\footnote{\url{https://www.iso.org/standard/22109.html}}) followed by its name in that language---alternatively concatenating its name first and then a language ID (see Section~\ref{sec:plain_generation});
    \item treat the selection of an identifier in a particular language as a latent variable (\textit{i.e.}, we let the model learn a conditional distribution of languages given the input and we marginalize over those---see Section~\ref{sec:marginalization}).
\end{enumerate}
All of these strategies define a different way we compute the underlining likelihood of our model. In Figure~\ref{fig:mGENRE} we show an outline of m\textsc{GENRE}\@\xspace. The following subsections will present detailed discussions of the above 3 strategies.

\subsection{Canonical entity representation} \label{sec:canonical}
Selecting a single textual identifier for each entity corresponds to choosing its name among all the available languages of that entity.
We employ the same data-driven selection heuristic as in~\citet{botha-etal-2020-entity}: for each entity $e$ we sort all its names $n_e^l$ for each language $l$ according to the number of mentions of $e$ in documents of language $l$. Then we take the name $n_e^l$ in the language $l$ that has the most mentions of $e$. In case of a tie, we select the language that has the most number of mentions across all entities (\textit{i.e.}, the language for which we have more training data).
Having a single identifier for each entity corresponds to having a 1-to-1 mapping between strings and entities. Thus, $\mathrm{score}_\theta(e|x) = p_\theta(n_e|x)$ where with $n_e$ we indicate the \textit{canonical} name for $e$. We train to maximize the scores for all our training data. A downside of this strategy is that most of the time, the model
cannot exploit the lexical overlap between the context and entity name since it has to translate it in the canonical one.

\subsection{Multilingual entity representation} \label{sec:plain_generation}
To accommodate the canonical representation issues, we can predict entity names in any language.
Concatenating a language ID $l$ and an entity name $n_e^l$ in different orders induces two alternative factorizations. We train maximizing the scores for all our training data: $\mathrm{score}_\theta(e|x) =$
\begin{equation}
\begin{cases}
p_\theta(l|x) \cdot p_\theta(n_e^l | x, l) & \text{for `lang+name'} \\
p_\theta(n_e^l| x) \cdot p_\theta(l|n_e^l,x) & \text{for `name+lang'} \\
\end{cases}
\end{equation}
The former corresponds to first predicting a distribution over languages and then predicting a title \textit{conditioning} on the language $l$ where the latter corresponds to the opposite.
Predicting the language first conditions the generation to a smaller set earlier during beam search (\textit{i.e.}, all names in a specific language). However, it might exclude some targets from the search too early if the beam size is too small.
Predicting the language last does not condition the generation of names in a particular language but it \textit{asks} the model to disambiguate the language of the generated name whenever it is ambiguous (\textit{i.e.}, when the same name in different languages corresponds to possibly different entities). Only 1.65\% of the entity names need to be disambiguated with the language.
In practice, we observe no difference in performance between the two approaches.
Both strategies define an N-to-1 mapping between textual identifiers and entities and then at test time we just use a lookup table to select the correct KB item. This N-to-1 mapping is an advantage compared to using canonical names because the model can predict in any available language and therefore exploit synergies between source and target language as well as avoiding translation.

\subsection{Marginalization} \label{sec:marginalization}
Differently from the plain generation strategies described above, we can treat the textual identifiers as a latent variable and express $\mathrm{score}_\theta(e|x)$ as the probability of the entity name in all languages and marginalizing over them: $\mathrm{score}_\theta(e|x)=$
\begin{equation}
    p_\theta(e|x) = \sum_{\langle l, n_e^l \rangle \in \mathcal{I}_e} p_\theta(n_e^l,l|x) \;.
\end{equation}
Marginalization exposes the model to all representations in all languages of the same entity and it requires a minor modification of the training procedure. Unfortunately, because computing $\mathrm{score}_\theta(e|x)$ requires a sum over all languages, both training, and inference with marginalization are more expensive than with simple generation (scaling linearly with the number of languages).
However, at least during inference, we can still apply BS to only marginalize using the top-k generations.
For this reason, we test this training strategy only on few languages but we evaluate marginalization even when training with the other generation strategies described above.

\subsection{Candidate selection} \label{sec:candidates}
Modern EL systems that employ cross-encoding between context and entities usually do not score all entities in a KB as it is too computational expensive~\citep{wu-etal-2020-scalable}. Instead, they first apply candidate selection (with a less expensive method first or just a non-parametric mention table) to reduce the number of entities before scoring. In our formulation, there is no need to do that since m\textsc{GENRE}\@\xspace uses Beam Search to efficiently generate. However, using candidates might help, and therefore, we also experiment with that. Scoring all candidates might not be always possible (sometimes there are thousands of candidates for a mention) and especially when using an N-to-1 mapping between textual identifiers there will be names to rank in all languages available for each candidate. Thus, when we use candidates, it is to constrain BS steps further, rather than to rank all of them.

Concretely, candidate selection is made with an alias table. Using the training data, we build a mention table where we record all entities indexed by the names used to refer to them.
Additionally, we also use Wikipedia titles as additional mentions (useful for entities that never appear as links), redirects, Wikidata labels, and aliases.

redpajama_set_name: RedPajamaArXiv


Cameron Emmanuel is a neo-classical composer and self-taught contemporary visual artist born and raised in Honolulu, Hawaii. The region's tropical beauty served as sources of inspiration while growing up and pursuing his artistic foundations in visual art and music. Between music concerts, art galleries, festivals, and online platforms, his work is exhibited internationally. He currently works out of his studio in San Francisco, where he lives at a cross between art and music.
His vibrating visual art explores complex intricate layers, permutations of theme and variation, and repetitive sampling from his own work. These imaginative compositions erupt with colorful layers, forming patterns and textures that provide viewers a glimpse into the multiverse. He is deeply obsessed with micro patterns and fractals observed in nature. At the center of Cameron Emmanuel's work are dimensions of imagination that evoke a visual adventure.
Be prepared to leave this world behind.
First, I densely layer mixed media paintings. Then, capture macro photos of intricate details in the paintings that are my "MacroScapes". Finally, I mirror those macros into digital art pieces of geometric patterns and fractals, resulting in my "Modern Mandalas". In keeping with music composition, the original paintings resemble visual scores. The macro photos are movements, and the Mandalas are measures. This three-step intermedia process grants me as much control as a composer, conductor, and musician would have, and explores transmutation of media and permutation of theme and variation.
These paintings are mini universes and are the foundation of my process. They are all about the micro details, as I create intricate and dynamic paintings by using a combination of mixed media such as iridescent, metallic, crushed glass, and UV-reactive paints that play with light. These are also experiments in chemistry, using various additives to vary viscosity and alter appearance. In many ways this is art as ritual, where time is dissolved into the paintings. Depending on the viewing angles and light conditions, the paintings change appearance and texture. I want to draw people's attention to the details, make them stop and really look, and connect them to their inner child with a sense of awe and wonder.
I find working with tiny details fascinating, so viewing the art at five feet away versus five inches away is very different. I want to capture the closer viewing experience, and I found a way. Through the eyes of a composer, I extract "movements" by enlarging details of my paintings using macro photography. This sampling of my own paintings takes viewers on a symphonic movement which explores texture, rhythm, and a serenade of color. An expression calling attention to the details rather than the whole, these photos express an inherent duality both terrestrial and celestial that arouses imagination. I find beauty in the ambiguous nature of the abstract form.
From painting, to photo, to digital art, my mandalas are derived from my macros photos and are rhythmic arrangements that merge my personal interests of music, art, and science that further explore sacred geometry, fractals, and symmetry. Although still abstract, It seems having symmetry and mirrored patterns helps the imagination to recognize familiar things. This symmetry creates a fun psychological phenomenon known as Pareidolia; where people interpret familiar patterns as faces, objects, or animals where none exists. I always like to ask, what do you see?
It is more than an image seen with our eyes; it is an actual moment in time that express both the microcosm and macrocosm simultaneously into a harmonious unity. Each as unique as a snowflake, these Mandalas capture a frozen moment that is a measure of the finite and a road to the infinite. Universes within universes, these are maps of the multiverse.
Cameron Emmanuel's style might be described as Neo-Classical. As a composer, He is inspired most by the Classical Masters. Often, he uses bold statements with lyrical melodies, colorful orchestration, and varied textures. Drawing his strengths from his slower movements, he express a varied palette of emotions from the darkest depression to the sublime and joyous.
Beginning his interest in Music Composition at the age of 13 in Hawaii, Cameron has composed an assortment of musical works for piano, small ensembles, orchestra, concert band, and symphonic orchestra. In 2012 he earned a degree at the University of Washington, Seattle, in music and ethnomusicology, specializing in the gamelan of Indonesia. He has since moved to San Francisco, where he continues to focus on his artistic career.
Various works have been performed by musicians at the University of Hawaii, including his "Nocturne II" by pianist Megumi Kurachi and "Brass Band on Piece 21" by the University's Brass Ensemble under direction of Grant Okamura. The Honolulu Symphony Orchestra performed his fourth movement of Symphony No.1 in collaboration with the University of Hawaii under direction of maestra Joan Laundry. In Seattle, the Seattle Symphonic Band premiered Tempest: for concert band under direction of Lauren Anderson in December of 2007. The Chinook Winds premiered Divertimento No. 1: for double wind quintet and bass clarinet in April of 2008 under direction of UW Doctorate student, Maggii Weitzel.
Attending the UW he has had many wonderful opportunities to work with the schools various ensembles and conductors, such as Ethan Chessin, Garry Brattin, Dr. Steven Morrison, Vu Nguyen, and director of bands Timothy Salzman. In his senior year at UW he worked with Salzman who led the Wind Ensemble in an arrangement of three Schubert Lieder songs arranged for band. For his senior thesis he completed an an original composition for the Seattle Pacific University Gamelan ensemble with Prof. Christina Sunardi.
He has had great opportunities abroad to bring some of his works to life, and selected works have been performed by musicians in various parts of the world. Conductor Garry Brattin (1965-2015) led the Yue Tao Concert Band in 2013 with "Tempest" in Taipei, Taiwan. In 2015 conductor Peter Wuttke led his double wind quintet with the premiere of "Divertimento No.2" and "Divertimento No.1" in Düsseldorf, Germany.


Start your new year on the right foot with some entries in our $200 Cash via PayPal giveaway!
What would you do if you won this? Bills? Save for a vacation?
Me? If I could enter and won, I would use it towards my yearly household insurance.
'Tis the season and good luck!
Pay my heat,water and phone bills !


text:
Luca Hänni — song chords
Luca Hänni
Luca Hänni, a singer from Switzerland, became famous in 2012 after winning in the ninth season of the German TV show "Germany Seeks the Superstar." The singer's debut album "My Name is Luca" was released after the end of the season and took top spots on the charts in Switzerland, Austria, and Germany. Luca Hänni was chosen to represent Switzerland at the Eurovision Song Contest 2019 in Tel Aviv.
Performer's official web site - lucamusic.ch
Songs with chords
She Got Me
Total songs with chords: 1
🎸 L
Categories: 1005, songs: 3738.

redpajama_set_name: RedPajamaCommonCrawl


text:
Q: dynamic variable names using matlab-tables HF.EH = EH.Return_EH;

I want to do the above statement. HF is a table, so is EH.
str = 'HE'
HF.(str) = (str).Return_EH;

The last line of code is not working. "HF.(str)" is fine. "(str).Return_EH" is not. How can I make "(str).Return_EH" work?
Right now my code looks like that:
EH = importraw('HFRX_Equity_Hedge_Index.csv', 'EH');
EMN = importraw('HFRX_Equity_Market_Neutral_Index.csv', 'EMN');
EDI = importraw('HFRX_Event_Driven_Index.csv', 'EDI');
FICA = importraw('HFRX_FI-Convertible_Arbitrage_Index.csv', 'FICA');
MCTA = importraw('HFRX_Macro_CTA_Index.csv', 'MCTA');
MAI= importraw('HFRX_Merger_Arbitrage_Index.csv', 'MAI');
RVA = importraw('HFRX_Relative_Value_Arbitrage_Index.csv', 'RVA');

% sanity check
if not(isequal(EH.Date, EMN.Date, EDI.Date, FICA.Date, MCTA.Date, MAI.Date, RVA.Date));
 error('Mismatch in Data');
end

% merge Hedgefund Data
HF = array2table(zeros(size(EH,1),8), 'VariableNames',{'Date', ...
                    'EH', 'EMN', 'EDI', 'FICA', 'MCTA', 'MAI', 'RVA'});
HF.Date = EH.Date;
HF.EH = EH.Return_EH;
HF.EMN = EMN.Return_EMN;
HF.EDI = EDI.Return_EDI;
HF.FICA = FICA.Return_FICA;
HF.MCTA = MCTA.Return_MCTA;
HF.MAI = MAI.Return_MAI;
HF.RVA = RVA.Return_RVA;

I thought there should be a better way.

A: Instead of looking for dynamic variable names, you can use a structure in the same way you use the table HF. First, you would import all data to one structure S:
S.EH = importraw('HFRX_Equity_Hedge_Index.csv', 'EH');
S.EMN = importraw('HFRX_Equity_Market_Neutral_Index.csv', 'EMN');
% and so on...

Then in your sanity check, and everywhere else in your code, you need to add S. before this tables. Now you can replace the last part with:
fld = fieldnames(S);
for k = 1:numel(fld)
    HF.(fld{k}) = S.(fld{k}).(['Return_' fld{k}]);
end

redpajama_set_name: RedPajamaStackExchange


In 1923, EBMUD was founded due to the rapid population growth and severe drought in the area. The district constructed Pardee Dam (finished in 1929) on the Mokelumne River in the Sierra Nevada, and a large steel pipe Mokelumne Aqueduct to transport the water from Pardee Reservoir across the Central Valley to the San Pablo Reservoir located in the hills of the East Bay region. In subsequent years, EBMUD constructed two additional aqueducts to distribute water to several other East Bay reservoirs. From the various large regional reservoirs, water is transported to treatment plants and delivered to local reservoirs and tanks, thence distributed by gravity to customers.
In the 1980s with federal grant funding, EBMUD undertook a major facility expansion to accommodate wet weather waste water overflow (i.e. the vastly increased system demand in the rainy season). This project took many years of construction for implementation, after the planning and Environmental Impact Statement phases.
In 1994, the EBMUD board of directors approved the Seismic Improvement Plan (SIP), a $189 million capital project designed to minimize damage and disruption in the event of a potential earthquake along the Hayward Fault. One of EBMUD's main water supply lines, the Claremont Tunnel, traverses the Hayward Fault, and the maximum credible earthquake along the fault could sever the tunnel, which was originally built in 1929. Some of the major projects included in SIP were the Southern Loop Pipeline, a new 11-mile (18 km) seismically-reinforced alternate route which would allow restoration of water service; and seismic upgrades to the Claremont Tunnel. The expected benefit of SIP was avoiding a potential $1.2 billion in lost revenue and damage resulting from a major earthquake.
In May 2008, EBMUD announced severe drought and austerity measures for its customers. With the easing of the drought, these measures were rescinded in 2010. EBMUD announced mandatory water rationing again in August 2014. The emergency regulations imposed during this prolonged drought were relaxed effective July 1, 2016, after the drought was declared officially ended.
As with other public entities, the District has significantly underfunded liabilities for legacy costs. These include $535 million for retirement and $89 million for retiree health.
EBMUD has several sources of revenue for both water and sewage treatment enterprises. These sources include the sale of water, hydroelectric power, system capacity charges, sewage treatment charges, connection fees, wet weather facilities charges, interest and property tax increments.
In 2007, the water system was anticipated to generate a total of $375.5 million in revenue. Water sales account for approximately 76 percent of the revenue, with System Capacity Charges generating an additional 7 percent in revenue. Property Tax Revenue is expected to generate an additional 5 percent of revenues, with interest, electric energy sales, reimbursements and other sources making up the remaining 12 percent of revenues.
Historically, 90 percent of the water used by EBMUD comes from the 577 square mile protected Mokelumne River watershed. EBMUD has water rights for up to 325 million U.S. gallons per day (MGD) (997 acre-feet) or a total of 364,000 acre-feet per year. In normal years, EBMUD reservoirs in the East Bay receive an additional 30,000 acre-feet of local water from runoff annually. In dry years, evaporation and other losses can total more than the local runoff. Runoff from the Mokelumne watershed is not sufficient to meet EBMUD customer needs in times of severe drought.
In April 2015, EBMUD declared a Stage 4 critical drought and has set a community-wide goal to reduce water use by 20%. To reach this goal, EBMUD has adopted new water rules that affect all customers and must supplement normal water supplies with water from additional sources, like 33,250 acre-feet from the Central Valley Project. EBMUD has enforced strict water restrictions in order to ensure all conservation measures are being taken. By the end of 2015, EBMUD was projected to have in storage 320,000 to 330,000 acre-feet of water.
On May 10, 2016, EBMUD declared an end to the drought emergency, as their reservoirs had more water than average. The board voted to relax many of the water conservation rules and the 25% surcharge, effective July 1, 2016. It announced that Pardee Reservoir had reached 100 percent of its capacity in January 2017 and had begun releasing excess water into Camanche Reservoir.
In 2009, environmental activists groups sued to block approval of an Environmental Impact Statement (EIS) for the Pardee Reservoir expansion proposal.[a] Following a two-year trial, Sacramento Superior Court Judge Timothy Frawley agreed with the plaintiffs' objections and rejected the EIS in 2011. His ruling stated that EBMUD had not adequately addressed all the potential environmental harms of the project. Specifically, Judge Frawley said the document completely omitted discussing a less destructive alternative - working with CCWD on expanding Los Vaqueros reservoir. Moreover, it neglected to discuss the impacts of: (1) flooding two whitewater runs (Electra and Middle Bar); (2) destroying the historic Middle Bar Bridge and (3) inundating a stand of black willow trees that were important to the culture of the native Miwok people. After reviewing the ruling, EBMUD announced in December, 2011, that it had dropped plans for a new dam on the Mokelumne River, and would work with CCWD on expanding Los Vaqueros.
^ Plaintiffs in the suit were:California Sportfishing Protection Alliance (CSPA), Friends of the River and the Foothill Conservancy.
^ "Case Study: Seismic Improvement Program, East Bay Municipal Utility District" (PDF). Earthquake Engineering Research Institute: Mitigation Center. Retrieved 21 December 2017.
^ "Water Supply Briefing April 28, 2015" (PDF). EBMUD. Retrieved 11 May 2015.
^ "Thankful for a wet winter." East Bay Municipal Utility District. 2017. Accessed February 16, 2017.
^ a b "EBMUD Has Yet Another Option Besides a New Mokelumne Dam." East Bay Express. June 3, 2009. Accessed July 5, 2017.
^ a b Shutes, Chris. "Major Victory for CSPA: East Bay MUD Abandons Plan To Raise Pardee Dam." California Sportfishing Protection Alliance. December 6, 2011. Accessed July 5, 2017.
^ a b Zito, Kelly. "EBMUD set back by Pardee Reservoir ruling." SF Gate. April 18, 2011. Accessed July 5, 2017.
This page was last edited on 13 January 2018, at 04:06 (UTC).


Image Title: Dash And Albert Indoor Outdoor Stair Runner Intended For Ideas 19. Post Title: Dash And Albert Runner. Filename: dash-and-albert-indoor-outdoor-stair-runner-intended-for-ideas-19.jpg. Image Dimension: 600 x 837 pixels. Images Format: jpg/jpeg. Publisher/Author: Kevon Sporer. Uploaded Date: Saturday - September 29th. 2018 04:59:40 AM. Category: Architecture. Image Source: interiordesign.net. Dash And Albert Stair Runners DIY Installation Inside Runner Plan 8. Advantages Of Dash And Albert Stair Runner Railing Stairs Inside Decor 6. Dash Albert Runner Thestark Co Within And Remodel 16. Dash And Albert Runner Stair Picture Pertaining To Prepare 15. Dash And Albert Stair Runner Www Com Throughout Designs 7. Stair Runners Rugs By Dash Albert Annie Selke Inside And Runner Designs 2. Bye Carpet Hello Dash And Albert Runner Gracious Style Blog In Prepare 3. Bye Carpet Hello Dash And Albert Runner Gracious Style Blog Intended For Remodel 10. Dash And Albert Runner Stair Installation Indoor Outdoor Diamond With Regard To Decor 12. Advantages Of Dash And Albert Stair Runner Railing Stairs Throughout Prepare 11.