text:
\section{Evaluation Setup}
\label{sec:evalSetup}

We evaluate the EL performance of \textit{MetaEL+} for a given set of ready-made EL tools. Since the previously introduced prediction task is an integral element of \textit{MetaEL+}, we also evaluate the prediction performance of the proposed supervised classifiers.

\subsection{Datasets}

We need datasets for which enough ground truth (GT) annotations are provided for training a supervised classifier. We considered the following three datasets, each one containing at least 1,000 training annotations: i) \textit{CONLL} (GT annotations for 1,393 Reuters articles \cite{hoffart2011robust}), ii) \textit{IITB} (GT annotations for 107 text documents drawn from popular web pages about sports, entertainment, science, technology, and health \cite{kulkarni2009collective}), iii) \textit{NEEL} (GT annotations for >9,000 tweets, provided by the 2016 NEEL challenge \cite{Microposts2016}).
Several other GT datasets have not been considered because of their very small size (e.g. \textit{ACE2004}, \textit{Aquaint}, \textit{KORE50}, \textit{Meij}, \textit{MSNBC}).
CONLL and NELL are already split into training and test sets. For IITB we considered the first 90\% of the provided annotations for training and the remaining 10\% for test (thus one can reproduce the results).
In all datasets, we do not consider GT annotations pointing to \textit{NULL} or \textit{OOKB} (out of knowledge base). Table \ref{tbl:datasetsStats} shows the number of documents and annotations per dataset used for training and test (considering only the documents having at least one GT annotation).

\begin{table}[H]
\vspace{-2mm}
\centering
\renewcommand{\arraystretch}{0.7}
\setlength{\tabcolsep}{4.5pt}
\caption{Ground truth datasets main statistics.}
\vspace{-4mm}
\label{tbl:datasetsStats}
\small
\begin{tabular}{rrrrr}
     \toprule
     Dataset     &  \makecell{\#Train docs} &  \makecell{\#Train annots} & \makecell{\#Test docs}  & \makecell{\#Test annots}\\
     \midrule
     CONLL      & 1,162   & 23,332   & 231    & 4,485 \\
     IITB       & 90      & 10,847   & 13     & 1,174 \\
	 NEEL       & 3,342   & 6,374    & 291    & 736   \\
	 \bottomrule
\end{tabular}
\vspace{-2.5mm}
\end{table}

\subsection{Entity Linking Tools}
\label{subsec:tools}

We deployed three popular state-of-the-art EL tools:
\textit{Ambiverse} (previously AIDA) \cite{hoffart2011robust},
\textit{Babelfy} \cite{moro2014entity}, and \textit{TagMe} \cite{ferragina2010tagme}.
These tools were selected because: i) they are end-to-end (ready-made) tools that can be easily used out-of-the-box, and ii) they are accessible through public APIs, thus one can directly use them. Moreover, they have been widely used in different contexts (each one having >400 citations). Other EL systems, including more recent ones that make use of neural models, have not been considered because they do not satisfy these criteria.
For Ambiverse, we used its default configuration. For Babelfy, we used the configuration suggested by the Babelfy developers.\footnote{The configuration is available at: \url{https://goo.gl/NHXVVQ}}
For TagMe we used its default configuration and a confidence threshold of 0.2 to filter out low quality annotations.

\begin{comment}
\begin{table}[!h]
\centering
\renewcommand{\arraystretch}{0.7}
\setlength{\tabcolsep}{4.5pt}
\caption{Performance of the used EL tools on CONLL \cite{hoffart2011robust}.}
\vspace{-3mm}
\label{tbl:perfEntLink}
\small
\begin{tabular}{p{1.6cm} p{1.6cm} p{1.2cm} p{0.8cm}}
     \toprule
     \textbf{System}     & \textbf{Precision (\%)} & \textbf{Recall (\%)}   & \textbf{F1 (\%)}    \\
     \midrule
     Ambiverse  & 80.7    & 64.7     & 71.8  \\
	 Babelfy    & 81.5    & 68.2     & 74.3  \\
	 TagMe      & 78.7    & 53.2     & 63.5   \\
	 \bottomrule
\end{tabular}
\vspace{-3mm}
\end{table}
\end{comment}

\subsection{Baseline and MetaEL+ Methods}
\label{subsec:baselines}

Since the objective of MetaEL is the selection of output from multiple EL tools for achieving a better performance, each of the used tools (Ambiverse, Babelfy and TagMe) is considered a different and naive baseline. In addition, considering the agreement of the tools on the provided entity (majority vote) or their overall performance in a ground truth dataset, are two other predictive baselines \cite{corcoglioniti2016microneel,si2005boosting}. As regards the \textit{MetaEL} problem per se, \cite{ruiz2015combining} proposes a weighted voting scheme which ranks the candidate entities by considering the performance of the tools on a so-called ranking corpus (CONLL).
The considered baselines are summarised below:

\begin{itemize}[noitemsep,nolistsep]
    \item Each considered EL system (\textbf{Ambiverse}, \textbf{Babelfy}, \textbf{TagMe}).
    \item \textbf{Random}: select one of the tools randomly.
    \item \textbf{Best System}: select the link provided by the system with the highest overall performance in the ground truth dataset.
    \item \textbf{Majority+Random}: select the link provided by the majority of the tools. If all tools provide a different link, a random one is selected.
    \item \textbf{Majority+Best}: select the link provided by the majority of the tools. If all tools provide a different link, the system with the highest overall performance is selected. This method is similar to the \textit{rule-based} method of \cite{corcoglioniti2016microneel}.
    \item \textbf{Weighted Voting}: the annotations are combined through the weighted voting scheme described in \cite{ruiz2015combining}. If the score is lower than the maximum precision for all annotators on the ranking corpus, the annotation is not considered.
    \item \textbf{Weighted Voting All}: the annotations are combined through the weighted voting scheme described in \cite{ruiz2015combining}, however without filtering out annotations with a score lower than the maximum precision for all annotators.
\end{itemize}

We compare the performance of the above mentioned baselines with the following two \textit{MetaEL+} approaches:

\begin{itemize}[noitemsep,nolistsep]
    \item \textbf{MetaEL+$_{\text{LOOSE}}$}: a multi-label binary relevance classifier (with RF as the base classifier) is used when more than one tool provide a link for the same mention. We use the implementation and default configuration of MEKA 1.9.3 \cite{MEKA}.
    When more than one system is predicted, we consider the \textit{prediction confidence} scores provided by the classifier for each class.
    In case of equal scores, we select the system with the highest overall performance in the training dataset.
    If only one EL system has recognised a mention, we trust it and assign the entity provided by this system.

    \item \textbf{MetaEL+$_{\text{STRICT}}$}: the same multi-label classifier from the MetaEL+$_{\text{LOOSE}}$ approach is used for cases where more than one tool provide a link for the same mention. However, this method is more selective: when a mention is recognised by only one EL system, a system-specific SMO binary classifier is used for predicting if the provided entity link is correct.
\end{itemize}

\subsection{Evaluation metrics}

\subsubsection{Evaluating EL performance}
We make use of the following metrics: \textbf{Precision (P)} (number of correctly disambiguated mentions divided by the number of recognised mentions), \textbf{Recall (R)} (number of correctly disambiguated mentions divided by the total number of not null annotations in the ground truth), and \textbf{F1 score (F1)} (harmonic mean of precision and recall).

\subsubsection{Evaluating the classification performance}
\label{subsubsec:evalMetricsClassPerf}

Evaluation metrics for multi-label classification are inherently different from those used in single-label classification (like binary or multi-class) \cite{tsoumakas2007multi}.
We report results for the following metrics: \textbf{Jaccard Index} (number of correctly predicted labels divided by the union of predicted and true labels), \textbf{Hamming Loss} (fraction of the wrong labels to the total number of labels), \textbf{Exact Match} (percentage of samples that have all their labels classified correctly), \textbf{Per-class Precision} (P), \textbf{Recall} (R) and \textbf{F1 score} (if $TL$ denotes the true set of labels for a given class and $PL$ the predicted set of labels for the same class, then $P = \frac{TL \cap PL}{PL}, R = \frac{TL \cap PL}{TL}$, and $F1 = \frac{ 2\cdot P \cdot R}{P+R}$).

Since, in our problem, the prediction of any of the tools that provides the correct entity is adequate, we also report the accuracy of the classifiers in each dataset when considering if the correct entity is provided by the predicted system. Based on this, we define \textbf{Real Prediction Accuracy} as the number of predictions for which the predicted system provides the correct entity divided by the total number of predictions.

Finally, for measuring the performance of the three binary classifiers used by the \textsc{STRICT} approach, we consider the \textbf{per-class P}, \textbf{R} and \textbf{F1 score} as well as the \textbf{macro-averaged F1 score}.

\section{Evaluation Results}
\label{sec:evalResults}

\subsection{Annotation Statistics and Upper Bound Performance}

\subsubsection{Annotation and agreement statistics}
Table \ref{tbl:annotStats} provides detailed statistics about the annotations of the test datasets using the three EL tools. These statistics can help us better understand the characteristics of the datasets and the behaviour of the considered tools.

\begin{table}[!h]
\centering
\renewcommand{\arraystretch}{0.7}
\setlength{\tabcolsep}{1.0pt}
\caption{Annotation statistics of the test datasets.}
\vspace{-4mm}
\label{tbl:annotStats}
\small
\begin{tabular}{p{4.3cm}rrrr}
\toprule
     & \textbf{CONLL}  & \textbf{IITB}   & \textbf{NEEL}  \\
\midrule
Total number of GT annotations:                                & 4,485   & 1,174     & 736   \\
Ambiverse annotations:                                         & 4,169   & 390       & 66   \\
Babelfy annotations:                                           & 9,578   & 868       & 246    \\
TagMe annotations:                                             & 4,626   & 355       & 801   \\
\midrule
GT mentions recognised by 0/3 tools:                           & 295 (6.6\%)     & 694  (59.1\%)   & 456 (62.0\%)  \\
 GT mentions recognised by 1/3 tools:                           & 468 (10.4\%)    & 227 (19.3\%)     & 225 (30.6\%) \\
\hspace{1.0mm} \textit{Correct entity is provided:}          & 337 (72\%)     & 63 (27.8\%)      & 141 (62.7\%)   \\
GT mentions recognised by 2/3 tools:                           & 1,251 (27.9\%)  & 125 (10.6\%)  & 43 (5.8\%) \\
\hspace{1.0mm} The 2 tools provide the same entity:              & 950  (75.9\%)   & 88 (70.4\%)         & 32 (74.4\%) \\
\hspace{1.0mm} The 2 tools provide different entities:           & 301 (24.1\%)     & 37 (29.6\%)         & 11 (25.6\%) \\
\hspace{1.0mm} \textit{Correct entity is provided:}     & 1,061 (84.8\%)  & 103 (82.4\%)           & 37 (86\%)  \\
GT mentions recognised by 3/3 tools:                           & 2,471 (55.1\%)  & 128 (10.9\%) & 12 (1.6\%)   \\
\hspace{1.0mm} 3/3 tools provide the same entity:  & 1,786 (72.3\%)   & 82 (64.1\%)       & 8 (66.7\%)    \\
\hspace{1.0mm} 2/3 tools provide the same entity:  & 618 (25\%)     & 37 (28.9\%)       & 3 (25\%)     \\
\hspace{1.0mm} Each tool provides a different entity: & 67 (2.7\%)      & 9 (7\%)        & 1 (8.3\%)    \\
\hspace{1.0mm} \textit{Correct entity is provided:}     & 2,314 (93.6\%)   &  119 (93\%)     & 12 (100\%)\\
\bottomrule
\end{tabular}
\vspace{-3mm}
\end{table}

The first four rows show the total number of GT annotations in each dataset and the number of annotations produced by each of the considered EL tools.
The next rows show the number of GT mentions recognised by zero, only one, two, or all three tools, as well as the number of mentions for which at least one of the tools provides the correct entity and the agreement of the tools in the provided entities.
We notice different patterns across the datasets. Concerning CONLL, for instance, we notice that the majority of GT mentions were recognised by all three tools (55.1\%), followed by mentions recognised by 2/3 tools (27.9\%).
On the contrary, for IITB and NEEL, we see that the majority of GT mentions were not recognised by at least one system (59\% and 62\%, respectively).
Based on these numbers, we expect a high improvement of recall when we combine the annotations of the three tools since a much larger number of GT annotations are expected to have been recognised by at least one system.
Moreover, we see that for a quite high percentage of mentions recognised by only one EL tool, the provided entity is not correct (28\% in CONLL, 72\% in IITB, 37\% in NEEL). Thus, an effective \textit{MetaEL} method focusing on high precision should avoid including these annotations in the unified set of entity annotations.

With respect to the agreement of the tools on the provided entities, we noticed that when more than one system provides an entity for the same mention, the tools usually agree on the entity.
Nevertheless, we notice that for CONLL there is a high percentage of GT mentions where the tools disagree and provide different entities (22\% of all GT annotations). This percentage is 7\% for IITB and around 2\% for NEEL. The problem with these two datasets (especially with NEEL) is that the percentage of GT mentions recognised by zero or only one system is very high (78\% for IITB and 93\% for NEEL), in contrast to CONLL where the percentage is only 17\%.
If we consider only the GT mentions for which at least two tools provide a link, then the percentage of mentions that need prediction is again high (26.5\% for CONLL, 32\% for IITB, and 27.3\% for NEEL).
For all these mentions, an effective \textit{MetaEL} method needs to predict the system that can provide the correct entity link.

The above analysis shows that there is (i) a high percentage of mentions for which the EL tools provide different entities, and (ii) a high percentage of mentions for which no EL tool provides the correct entity. This means that predicting the system to consider or the correctness of an annotation can significantly improve the overall EL performance.

\subsubsection{Upper bound performance}
\label{sbusubsec:ubp}

Given the GT of each dataset, we can compute the performance of an ideal \textit{MetaEL} system that always makes a correct prediction (and thus no other method can provide better results).
The first row in Table  \ref{tbl:ELperformance}
shows the upper bound performance for each of the considered datasets. Comparing the upper bound performance for CONLL with the performance of the three individual tools on the same dataset (rows 2-4 in Table \ref{tbl:ELperformance}), we notice that \textit{MetaEL} can highly increase the F1 score from 74.3\% (of Babelfy, the top performing system) to 90.6\%, i.e., >15 percentage points (or 22\% increment).
With respect to the other datasets, we see that the F1 score of the upper bound performance is relatively low. As we will see below, the reason is the low recall achieved by all tools. Nevertheless, the F1 score of the upper bound performance is much higher than that of the top performing individual system in each case (33\% increment for IITB and 18\% for NEEL).
These results provide a good motivation for an effective \textit{MetaEL} method that can achieve a high performance as close to the upper bound performance as possible.

\vspace{-2mm}
\subsection{Entity Linking Performance}
\label{subsec:metaelPerf}
Table \ref{tbl:ELperformance} shows the EL performance of all approaches on the different datasets. The first row shows the upper bound performance and the next three rows the performance of the individual EL tools. The next six rows show the performance of the six baseline methods and the last two rows the performance of our two \textit{MetaEL+} methods.

\begin{table*}[]
\vspace{-3mm}
\centering
\caption{Entity linking performance.}
\vspace{-4mm}
\renewcommand{\arraystretch}{0.7}
\setlength{\tabcolsep}{4.5pt}
\small
\label{tbl:ELperformance}
\begin{tabular}{@{}lccc|ccc|ccc@{}} \hline
    \multirow{2}{*}{\textsc{\textbf{Method}}}
    & \multicolumn{3}{c}{\textsc{\textbf{CONLL-Test}}}
    & \multicolumn{3}{c}{\textsc{\textbf{IITB-Test}}}
    & \multicolumn{3}{c}{\textsc{\textbf{NEEL-Test}}} \\
 \multicolumn{1}{c}{}  & P (\%) & R (\%) & F1 (\%) & P (\%) & R (\%) & F1 (\%) & P (\%) & R (\%) & F1 (\%) \\ \hline

 \textsc{Upper bound} &
 100.0 & 82.8 & 90.6 &
 100.0 & 24.3 & 39.1 &
 100.0 & 25.7 & 40.9 \\ \hline \hline

  \textsc{Ambiverse} &
  80.7 & 64.7 & 71.8 &
  \textbf{85.2} & 17.7 & 29.3 &
  \textbf{76.6} & 4.9 & 9.2 \\

  \textsc{Babelfy} &
  81.5 & 68.2 & 74.3 &
  42.7 & 13.7 & 20.8 &
  64.4 & 3.9 & 7.4 \\

 \textsc{TagMe} &
 78.7 & 53.2 & 63.5 &
 72.3 & 14.9 & 24.7 &
 67.5 & 23.4 & 34.7 \\ \hline

  \textsc{Random} &
  79.3 & 74.1 & 76.7 &
  52.7 & 21.6 & 30.6 &
  64.3 & 24.5 & 35.4 \\

  \textsc{Best System} 	 &
  80.3 & 75.0 & 77.5 &
  57.9 & \textbf{23.7}& 33.6 &
  65.7 & \textbf{25.0} & \textbf{36.2} \\

 \textsc{Majority+Random}	 &
 80.8 & 75.5 & 78.0 &
 54.7 & 22.4 & 31.8 &
 65.4 & 24.9 & 36.0 \\

 \textsc{Majority+Best}  	 &
 80.5 & 75.3 & 77.8 &
 57.7 & 23.6 & 33.5 &
 65.7 & \textbf{25.0} & \textbf{36.2} \\

  \textsc{Weighted Voting}  	 &
  80.8 & 72.5  & 76.4 &
  44.8 & 17.3  & 25.0 &
  63.5 & 22.7  & 33.4 \\

  \textsc{Weighted Voting All}  	 &
  80.3 & 75.0  & 77.5 &
  48.0 & 19.4  & 27.6  &
  65.7 & \textbf{25.0}  & \textbf{36.2} \\ \hline

  \textsc{MetaEL+$_{\text{LOOSE}}$} 	 &
   84.8 & \textbf{79.2} & \textbf{81.9} &
   57.7 & 23.6 & 33.5 &
   65.7 & \textbf{25.0} & \textbf{36.2} \\

  \textsc{MetaEL+$_{\text{STRICT}}$} 	 &
  \textbf{86.6} &  75.2 & 80.5 &
  84.8 &  22.3 & \textbf{35.3} &
  73.0 &  9.9  & 17.5  \\

  \hline
\end{tabular}
\vspace{-3mm}
\end{table*}

To calculate the statistical significance of our results, we divided the test set of each dataset into 20 disjoint splits of equal number of annotations, and computed the F1 score on each split for each method (similar to the approach in \cite{fang2014entity}). Two-tail paired t-test was then applied to determine if the F1 scores of our methods and the baselines are significantly different.

First, we notice that the performance of the individual EL tools varies across datasets. As regards CONLL, Babelfy is the top performing tool and TagMe the tool with the worst performance (in terms of F1 score). For IITB, Ambiverse is the top performing tool and Babelfy the worst one. For NEEL, TagMe is the tool with the best performance and Babelfy the one with the worst performance.
These results validate our motivation that the performance of EL systems varies across datasets.

Regarding the performance of the proposed MetaEL+ approaches, we notice that our \textsc{LOOSE} approach achieves the highest F1 score in CONLL (the largest and most reliable dataset), outperforming the top performing individual system by 10\% (from 74.3\% to 81.9\%) and the top performing baseline by 5\% (from 78\% to 81.9\%). In more detail, recall of the top performing EL system (\textsc{Babelfy}) is improved from 68.2\% to 79.2\% (very close to the upper bound performance) and at the same time precision is improved from 81.5\% to 84.8\%. This is very promising given that, usually, improvement in recall affects precision negatively.
With respect to the baseline methods, recall of the top performing baseline (\textsc{Majority+Random}) is improved from 75.5\% to 79.2\% and precision from 80.8\% to 84.8\%. All these improvements are statistically significant for $\alpha$-level = 0.05.
We also see that, with a drop of recall to 75.2\%, precision can be further improved to 86.6\% using the \textsc{STRICT} approach. Here we would expect a higher improvement of precision, which means that the binary classifiers are not probably very effective in distinguishing \textit{true} from \textit{false} instances (this hypothesis is validated below).

In IITB, our \textsc{MetaEL+$_{\text{STRICT}}$} approach achieves the highest F1 score, outperforming the top performing EL tool (\textsc{Ambiverse}) by 20.5\% and the top performing baseline by 5\%.
Our method combines a high recall (compared to that of the individual systems) with a very high precision (84.8\%). Precision, in particular, is improved compared to the best baseline (\textsc{Best System}) by 46.5\% while recall slightly drops from 23.7\% to 22.3\%.

Finally, in NELL we notice that our \textsc{LOOSE} approach and four of the baseline systems achieve the same performance. This is not surprising given the very small number of cases that need prediction in this dataset (cf. Table \ref{tbl:annotStats}).
As regards the \textsc{STRICT} approach, we see that it highly improves precision from 65.7\% (of the top performing baseline) to 73\%, however with the cost of a high drop of recall (from 25\% to almost 10\%).

These results demonstrate that the proposed \textit{MetaEL+} methods can significantly improve the performance of the individual systems and achieve results that are even competitive to
recent EL systems that make use of neural models, like \cite{cao2018neural} and \cite{kolitsas2018end} that report an F1 score of 80\% and 82.4\%, respectively, on the \textsc{CONLL} dataset.

\subsection{Prediction Performance}
\label{subsec:predPerformance}

\subsubsection{Multi-label classification}
Table \ref{tbl:multilabelPerf} shows the prediction performance of our multi-label classifier.
We see that \textit{Jaccard Index} (ratio of correctly predicted labels) is high for CONLL (50.5\%) and IITB (58.7\%) but low for NEEL (36.5\%).
\textit{Hamming Loss} (ratio of wrong labels) ranges from 26.9\% (for IITB) to 41.3\% (for CONLL). With respect to the most strict metric \textit{Exact Match}, the score is 36.1\% for CONLL, 54.0\% for IITB, and 30\% for NEEL. In general, we see that the classification performance is very good for IITB and satisfactory for CONLL.
As we have already stressed (cf. Sect. \ref{subsubsec:evalMetricsClassPerf}), these metrics evaluate the correct prediction of all class labels per instance.
The \textit{real prediction accuracy} (last row of Table \ref{tbl:multilabelPerf}) shows the classification performance when considering if the correct entity is provided by the predicted system.
The score is more than $90\%$ for CONLL and IITB, and almost 70\% for NEEL. These results demonstrate the high performance of our multi-label classifier.

Looking now at the per-class performance, we see that for CONLL, the class label \textit{Babelfy} achieves the highest F1 score (69.8\%) while the \textit{TagMe} class has the lowest score (61.2\%).
On the contrary, in IITB the \textit{Ambiverse} class achieves the highest F1 score (60.5\%), due to its very high precision (90.0\%), and \textit{TagMe} the lowest (45.1\%).
In NEEL, the highest F1 score is again achieved by the \textit{Ambiverse} class (45.0\%), however the lowest by the \textit{Babelfy} class (31.9\%).
These results show that there is no class for which the classifiers have a consistent high performance.

\begin{table}[]
\centering
\centering
 \caption{Performance of multi-label classification}
\vspace{-4mm}
\renewcommand{\arraystretch}{0.6}
\setlength{\tabcolsep}{4.5pt}
\small
\label{tbl:multilabelPerf}
\begin{tabular}{@{}lcccc@{}}
\toprule
 \textbf{Evaluation metric}  & \textbf{CONLL} & \textbf{IITB} & \textbf{NEEL} \\ \midrule

  Jaccard Index (\%) &
  50.5 & 58.7  & 36.5  \\

  Hamming Loss (\%) &
  41.3 &  26.9  & 29.9  \\

  Exact Match (\%) &
  36.1 & 54.0  &	30.0  \\ \midrule

  Precision (\%) of Ambiverse class &
  76.0 & 90.0  & 45.7  \\

  Precision (\%) of Babelfy class &
  81.8 & 86.4  & 23.1 \\

  Precision (\%) of TagMe class &
  60.6 & 62.0 & 68.2 \\ \midrule

  Recall (\%) of Ambiverse class &
  60.4 &  46.0 & 44.4   \\

  Recall (\%) of Babelfy class &
  60.8 &  35.4 &    51.7  \\

  Recall (\%) of TagMe class &
  61.9 & 35.4 &  26.2 \\ \midrule

  F1 (\%) of Ambiverse class &
  67.3 & 60.5 & 45.0  \\

  F1 (\%) of Babelfy class &
  69.8 & 50.2 &  31.9  \\

  F1 (\%) of TagMe class &
  61.2 & 45.1 &  37.9 \\ \midrule
  \textbf{Real Prediction Accuracy (\%)} &  91.1 & 95.9  & 69.6 \\
  \bottomrule
\end{tabular}
\vspace{-5mm}
\end{table}

\subsubsection{Binary classification}
Table \ref{tbl:binClassPerf} shows the performance of the three binary classifiers used by the \textsc{MetaEL+$_{\text{STRICT}}$} method. First, we should highlight that the class distribution is very unbalanced. On average, around 78\% of the annotations are correct (\textit{true} class) and 22\% are wrong (\textit{false} class). This means that the \textit{false} class is underrepresented, which makes the classification problem harder.

As expected, we notice that precision is very high for the majority \textit{true} class in almost all cases, while recall is high for the minority \textit{false} class. In CONLL, for example, precision of the \textit{true} class ranges from 88.6\% (TagMe classifier) to 91.5\% (Babelfy), while that of the \textit{false} class ranges from 25.5\% (Ambiverse) to 30.5\% (TagMe). On the contrary, recall of the \textit{true} class ranges from 45\% (Ambiverse classifier) to 54.2\% (TagMe) and of the \textit{false} class from 40.4\% (Babelfy) to 79\% (Ambiverse).
Looking at the macro-averaged F1 scores, we notice that their performance is close to 50\% in almost all cases.
TagMe classifier has the best performance in the two largest datasets (CONLL, IITB), however it has the worst performance in NEEL.
It is evident from these results that there is much room for further improvement for binary classification.

\begin{table}
\centering
\caption{Performance of binary classification.}
\vspace{-4mm}
\renewcommand{\arraystretch}{0.6}
\setlength{\tabcolsep}{4.5pt}
\small
\label{tbl:binClassPerf}
\begin{tabular}{@{}lcccc@{}}
\toprule
 \textbf{Evaluation metric}  & \textbf{CONLL} & \textbf{IITB} & \textbf{NEEL} \\ \midrule

  Ambiverse - Precision (\%) of \textit{true} class     &  89.9 & 98.2 &  89.5  \\
  Ambiverse - Precision (\%) of \textit{false} class    &  25.5 & 21.4 &  31.0  \\ \midrule
  Ambiverse - Recall (\%) of \textit{true} class        &  45.0 & 46.6 &  45.9  \\
  Ambiverse - Recall (\%) of \textit{false} class       &  79.0 & 94.4 &  81.8 \\ \midrule
  Ambiverse - F1 (\%) of \textit{true} class            &  59.9 & 63.2 &  60.7  \\
  Ambiverse - F1 (\%) of \textit{false} class           &  38.6 & 34.9 &  45.0  \\
  \midrule \midrule

  Babelfy - Precision (\%) of \textit{true} class     &  91.5 & 96.5 &  93.8  \\
  Babelfy - Precision (\%) of \textit{false} class    &  27.2 & 66.3 &  51.7  \\ \midrule
  Babelfy - Recall (\%) of \textit{true} class        &  52.2 & 30.2 &  51.7 \\
  Babelfy - Recall (\%) of \textit{false} class       &  40.4 & 79.5 &  66.7  \\ \midrule
  Babelfy - F1 (\%) of \textit{true} class            &  66.5 & 46.0 &  66.7  \\
  Babelfy - F1 (\%) of \textit{false} class           &  38.6 & 34.9 &  45.0  \\
  \midrule \midrule

  TagMe - Precision (\%) of \textit{true} class     &  88.6 & 92.9 &  66.7 \\
  TagMe - Precision (\%) of \textit{false} class    &  30.5 & 35.5 &  32.3 \\ \midrule
  TagMe - Recall (\%) of \textit{true} class        &  54.2 & 33.2 &  20.9 \\
  TagMe - Recall (\%) of \textit{false} class       &  74.3 & 93.5 &  78.3 \\ \midrule
  TagMe - F1 (\%) of \textit{true} class            &  67.3 & 48.9 &  31.9  \\
  TagMe - F1 (\%) of \textit{false} class           &  43.3 & 51.4 &  45.8  \\
  \midrule \midrule
   Ambiverse - Macro-averaged F1 (\%) & 49.3  & 49.1 & 52.9   \\
   Babelfy - Macro-averaged F1 (\%)   & 52.6  & 40.5 & 55.9   \\
   TagMe - Macro-averaged F1 (\%)     & 55.3  & 50.2 & 38.9   \\
  \bottomrule
\end{tabular}
\vspace{-4mm}
\end{table}

\subsection{Feature Analysis}

Table \ref{tbl:featureAnalysis} shows the EL performance for different combinations of features when considering the largest ground truth dataset (CONLL) and the best performing \textit{MetaEL+} method (\textsc{MetaEL+$_{\text{LOOSE}}$}).

With respect to the categories of features, we notice that the best performance is achieved when all categories are combined, which means that all contribute on achieving a high performance.
Regarding each individual category, we see that the \textit{surface form-based} features have the best performance, achieving an F1 score of 80.4\%. The \textit{mention-based} and \textit{document-based} features achieve 77.6\% and 78.2\%, respectively.
The best pair of feature categories is the \textit{surface form-based} and \textit{document-based} (81\% F1) and the worst pair is the \textit{mention-based} and \textit{document-based} (77.6\% F1). These results show that the \textit{surface form-based} features have the highest contribution on achieving a good EL performance, and the \textit{mention-based} features the lowest contribution.

Regarding the influence of each individual feature, we notice that $s_{ratio}$  (surface form's ratio of correct disambiguations per EL system) has the highest effect when we exclude it, dropping the F1 score from 81.9\% to 81\%. The second most influential feature is $m_{pos}$ dropping the F1 score to 81.1\%, which means that the mention's position in the document is a good indicator for the system that provides the correct entity.

\begin{table}[h]
\vspace{-3mm}
\small
\centering
\renewcommand{\arraystretch}{0.6}
\setlength{\tabcolsep}{4.5pt}
\caption{Effectiveness of different feature combination using \textsc{MetaEL+$_{{\text{LOOSE}}}$} on CONLL.}
\vspace{-4mm}
\label{tbl:featureAnalysis}
\footnotesize
\begin{tabular}{p{4.5cm}ccc}
     \toprule
     \textbf{Features} & \textbf{P (\%)} & \textbf{R (\%)}    & \textbf{F1 (\%)} \\
     \midrule
All features	&	\textbf{84.8}	&	\textbf{79.2}	&	\textbf{81.9}	\\	\midrule
Only surface form-based	&	83.2	&	77.7	&	80.4	\\
Only mention-based	&	80.3	&	75.1	&	77.6	\\
Only document-based	&	80.9	&	75.6	&	78.2	\\	\midrule
Surface form-based + mention-based	&	83.5	&	78.0	&	80.7	\\
Surface form-based + document-based	&	83.8	&	78.3	&	81.0	\\
Mention-based + document-based	&	80.3	&	75.1	&	77.6	\\	\midrule
All features except $s_{words}$	&	84.4	&	78.9	&	81.5	\\
All features except $s_{f}$	&	84.0	&	78.5	&	81.2	\\
All features except $s_{df}$	&	84.5	&	78.9	&	81.6	\\
All features except $s_{cand}$	&	84.4	&	78.9	&	81.5	\\
All features except $s_{corr}$	&	84.3	&	78.8	&	81.5	\\
All features except $s_{ratio}$	&	83.8	&	78.3	&	81.0	\\
All features except $m_{pos}$	&	83.9	&	78.4	&	81.1	\\
All features except $m_{sent}$	&	84.1	&	78.6	&	81.3	\\
All features except $d_{words}$	&	84.5	&	78.9	&	81.6	\\
All features except $d_{ents}$	&	84.0	&	78.5	&	81.2	\\	 	\bottomrule
\end{tabular}
\vspace{-6mm}
\end{table}

\subsection{Synopsis and Limitations}

The evaluation results can be summarised as follows:
\begin{itemize}[noitemsep,nolistsep]
    \item Combining multiple EL tools through a \textit{MetaEL} approach can achieve a significantly better EL performance than individual systems in isolation.

    \item The proposed supervised ensemble approach (\textit{MetaEL+}) significantly outperforms the individual EL tools and six baseline methods in the largest and most reliable datasets.

    \item A \textsc{STRICT} \textit{MetaEL+} method which predicts if the entity provided by an EL system is correct can further improve precision without significantly affecting recall.

    \item The proposed multi-label classifier
    achieves a prediction accuracy of >90\% in the two largest datasets of our evaluation (CONLL and IITB), demonstrating its effectiveness.

    \item The proposed binary classifiers achieve a relatively low accuracy (F1 score $\approx$ 50\%), showing that there is much room for improvement of the \textsc{STRICT} \textit{MetaEL+} method.

    \item All three categories of features contribute to achieving the highest performance. With respect to the individual features, $s_{ratio}$ (surface form's ratio of correct disambiguations by each EL system) and $m_{pos}$ (mention's normalised position in the document) seem to be the most influential features.
\end{itemize}

Limitations of our work are mainly concerned with (i) the limited performance of the binary classifiers in the \textsc{STRICT} approach, and (ii) the need of corpus-specific training data.

\section{Introduction}
\label{sec:introduction}

Entity linking (EL), or named entity recognition and disambiguation (NERD), is the task of determining the identity of entity mentions in text, thus linking a mention to an entity in a reference Knowledge Base (KB) like Wikipedia \cite{shen2015entity}.
For example, in the sentence \lq\lq{}{}Jordan played for the Wizards\rq\rq{}{}, a typical EL system would link the term \lq\lq{}{}Jordan\rq\rq{}{} to the Wikipedia page of the basketball player \textit{Michael Jordan} and the term \lq\lq{}{}Wizards\rq\rq{}{} to the Wikipedia page of the USA basketball team \textit{Washington Wizards}.

EL is a crucial task of relevance for a wide variety of applications, such as information retrieval \cite{raviv2016document}, document classification \cite{ni2016semantic},
or topic modelling \cite{chen2016probabilistic}.
Usually, high precision and recall are required if EL results are to have a positive impact on any such application.

However, EL remains a challenging task. EL systems differ along multiple dimensions and are evaluated over different datasets \cite{shen2015entity}, while their performance varies significantly across domains and corpora \cite{roder2017gerbil}. For instance, evaluations using the GERBIL benchmark \cite{roder2017gerbil} have shown that the performance of EL systems is highly affected by the characteristics of the datasets, such as the number of entities per document, the average document length, or the salient entity types \cite{usbeck2015evaluating}.
Thus, general-purpose EL remains a challenging task, where no single system has yet emerged as de-facto-standard across corpora and EL scenarios.

EL performance also varies strongly on each individual mention in the same corpus. As we will show in our evaluation (Table \ref{tbl:ELperformance}), the F1 score of three established  EL systems (TagMe, Ambiverse, Babelfy) on the popular CONLL dataset \cite{hoffart2011robust} ranges between 63.5\% - 74.3\% with an upper bound performance of 90.6\% when selecting the most correct outputs of all three systems.
This underlines that selecting the EL system on a per mention-basis rather than for a particular corpus, can significantly increase the EL performance.
However, the selection of the most suitable system for a given mention remains a challenge.
Prior works have shown that mentions which are difficult to link often share common characteristics \cite{fafalios2019SameButDifferent,hoffart2012kore}, which include ambiguity, indicated by a large number of candidates, mentions of long-tail entities which are not well represented in reference KBs, or mentions recognised in short documents with very limited context information.

Drawing on these observations, we argue that effective features can be derived from the corpus, the mention or the surface form to be linked, in order to predict the best-performing EL system on a per-mention-basis using supervised models.
In this work we introduce an ensemble learning approach towards exploiting the EL capabilities of a set of ready-made EL systems not only for improving recall, but also to improve precision by predicting the most correct EL system considering the particular characteristics of each particular mention.
We focus on exploiting ready-made (end-to-end) EL systems that are used as {\em black-box} systems using their default (suggested) configuration and without any corpus-specific training or tuning, because such systems are widely used in different contexts by also non-expert users.

We apply our approach to three established EL datasets and demonstrate significant performance improvements compared to both the individual EL systems and six baseline strategies.
Specifically, when considering the largest dataset (CONLL), our ensemble-based method significantly outperforms the best performing individual EL system by 10\% of F1 score, as well as the top performing baseline by 5\%.

In a nutshell, we make the following contributions:

\begin{itemize}
    \item We introduce the problem and a novel approach towards \textit{Meta Entity Linking}, in short \textit{MetaEL}, where outputs of multiple end-to-end EL systems are combined using an ensemble learning method for providing an improved set of entity links for a given corpus.
    \item  We propose a diverse set of features which give suitable signals for predicting the EL system that can provide the correct link for a given mention, and build supervised classifiers which are used as part of an automated \textit{MetaEL} pipeline.
    \item Using existing ground truth datasets and a set of three established and ready-made EL systems, we first provide detailed annotation and agreement statistics which demonstrate the potential performance improvement that an effective \textit{MetaEL} method can provide.
    Then, we report the resulting EL performance gain of the proposed supervised approach as well as evaluation results on the prediction task per se and the importance of the devised features, discussing also the limitations of our approach.
\end{itemize}

The rest of the paper is organised as follows:
Section \ref{sec:problem} formulates the \textit{MetaEL} problem and provides an overview of our approach.
Section \ref{sec:classification} details how supervised classification can be used for the problem at hand.
Section \ref{sec:evalSetup} describes the evaluation setup.
Section \ref{sec:evalResults} reports the evaluation results.
Section \ref{sec:rw} discusses related works and the difference of our approach.
Finally, Section \ref{sec:conclusion} concludes the paper and discusses interesting directions for future research.

\section{Conclusions and Future Work}
\label{sec:conclusion}

We have argued that the performance of entity linking (EL) on a given corpus may be optimised by combining the results of distinct EL tools. To this end, we introduced a novel approach towards \textit{Meta Entity Linking (MetaEL)} where outputs of multiple end-to-end EL tools are unified on a per-mention basis through an ensemble learning approach. We model the problem as a supervised classification task and provide a rich set of features that can be used within a supervised classifier for predicting the EL system that can provide the correct entity link for a given mention.

Using existing ground truth datasets and three EL tools, we compared the performance of the proposed models with each individual EL tool and with six baseline methods. The results show that, considering the largest ground truth dataset (CONLL), our multi-label classifier significantly outperforms the F1 score of both the best performing individual EL system (by 10\%) and the best baseline (by 5\%). Using binary classification for cases where a mention is recognised by only one EL system, a more selective (\textsc{STRICT}) approach that predicts the correctness of the provided entity link can further improve precision without significantly affecting recall.
Results on the performance of the prediction tasks per se demonstrated the effectiveness of the proposed multi-label classifier. Finally, an extensive feature analysis showed that all the proposed features contribute on achieving a high EL performance.

Given the promising results of our experiments, in the future we plan to extensively evaluate the performance gain of MetaEL using different number and combinations of EL tools, including more recent tools that make use of neural models. This will provide a better understanding of the circumstances under which MetaEL has a significant effect in the EL performance.
We also intend to study distantly supervised approaches where weakly labelled training data are automatically generated based on heuristics, aiming at solving the problem of obtaining corpus-specific training data. Finally, we plan to investigate the applicability of more advanced models for the binary classification task, in order to improve its (relatively low) performance.

\begin{acks}
This work was partially supported by CNPq (Brazilian National Council for Scientific and Technological Development) under grant GDE No. 203268/2014-8.
\end{acks}

\bibliographystyle{ACM-Reference-Format}

\section{MetaEL+}
\label{sec:classification}

This section describes the \textit{features}, \textit{classifiers} and \textit{labelling} methods used by the proposed \textit{MetaEL+} approach.

\subsection{Features}
\label{subsec:features}

We propose a set of features that can be easily computed for arbitrary corpora, i.e., we are not interested in features that, for example, require special metadata information about the documents. Inspired by related works on EL which study different factors that affect the performance of EL systems \cite{shen2015entity,roder2017gerbil}, as well as by the observed characteristics of mentions that fail to be disambiguated correctly, we consider features of the following categories:
i) \textit{surface form-based} (features related to the word or sequence of words representing an entity),
ii) \textit{mention-based} (features related to the mention recognised in a document, in a specific position), and
iii) \textit{document-based} (features related to the document containing the mention).
Below we detail each one of them.

\vspace{1.5mm} \noindent
{\em \large Surface Form-based Features \normalsize}
\vspace{0.5mm} \noindent

\noindent
\textbf{Number of words ($s_{words}$):} the number of surface form's words. Our intuition is that an EL system may perform better/worse on unigram surface forms that are usually more ambiguous than surface forms with more than one word.

\noindent
\textbf{Frequency ($s_{f}$):} the number of surface form's occurrences within the document. Our intuition is that more occurrences implies that the document topic is closely related to the surface form, indicating more representative context to facilitate its disambiguation.

\noindent
\textbf{Document frequency ($s_{df}$):} the number of documents in the corpus $D$ containing at least one occurrence of the surface form. Higher value implies popularity of the surface form, suggesting that more context is available about it which can facilitate its disambiguation by EL systems.

\noindent
\textbf{Number of candidate entities ($s_{cand}$):} the number of candidate entities in the reference KB (obtained by exploiting Wikipedia hyperlinks with anchor texts pointing to entities). We anticipate that individual EL systems may perform better/worse on ambiguous mentions having a high number of candidate entities.

\noindent
\textbf{Surface form's correct disambiguations per EL system ($s_{corr}$):} number of times the surface form has been disambiguated correctly by a specific EL system on the given training dataset.
Our intuition is that an EL system which has disambiguated correctly a particular occurrence of a surface form is more likely to disambiguate correctly a different occurrence of the same term.

\noindent
\textbf{Surface form's ratio of correct disambiguations per EL system ($s_{ratio}$):}  ratio of times the surface form has been disambiguated correctly by a specific EL system on the given training dataset. It is computed as the number of correct disambiguations divided by the sum of correct and wrong disambiguations.
Similarly to the previous feature, our intuition is that an EL system which performed well on a number of occurrences of a particular surface form will perform well on the same term in the future.

\vspace{1.5mm} \noindent
{\em \large Mention-based Features \normalsize}
\vspace{0.5mm}

\noindent
\textbf{Mention's normalised position ($m_{pos}$):} the mention's normalised position in the document, computed as the number of characters from the start of the document divided by the total number of characters in the document. Entities appearing early in the document are usually salient and representative for the document, indicating more representative context to facilitate their disambiguation.

\noindent
\textbf{Mention's sentence size ($m_{sent}$):} the number of characters of the sentence containing the mention, specifically the length of the text between two punctuation marks containing the mention (considering only the punctuation marks ".", "!", "?", ";"). Whereas an EL system may exploit the sentence containing the mention for disambiguating the entity, larger sentences may indicate more representative context for a particular mention.

\vspace{1.5mm} \noindent
{\em \large Document-based Features \normalsize}
\vspace{0.5mm}

\noindent
\textbf{Document size ($d_{words}$):} the number of words of the document containing the mention. We anticipate that the document length may provide signals for EL system performance with some approaches being able to deal better with short documents (containing more concise but less context), while others with longer documents.

\noindent
\textbf{Document's recognised entities ($d_{ents}$):} the total number of entities recognised in the document containing the mention. Given that EL systems tend to jointly disambiguate entities, some EL systems may perform better in the presence of a larger amount of recognised entities.

\subsection{Classifiers}
\label{subsec:models}

Since more than one EL system can provide the correct entity link for a recognised entity mention, we model the problem as a \textit{multi-label classification} task \cite{tsoumakas2007multi} where multiple labels (systems) may be assigned to each instance (entity mention).
We experimented with a large number of different methods using the MEKA framework \cite{MEKA} (an open source implementation of several methods for multi-label classification), trying also different base classifiers for each method, including Naive Bayes (NB), Logistic Regression (LR), J48, Random Forest (RF), and Sequential Minimal Optimisation (SMO).
In our evaluation (Sect. \ref{sec:evalResults}), we report results only for the top performing method: \textit{Binary Relevance} using {\em RF} as the base classifier.

As regards the case where only one EL system has recognised an entity mention $m$, a STRICT approach (as described in Sect. \ref{sec:problem}) needs to predict if the provided entity link is correct. For this, we need $n$ binary classification models, one for each considered EL system $(l_1, .. , l_n)$, where the class label is either \textit{true} (the EL system provides the correct entity link for $m$), or \textit{false} (the EL system does not provide the correct entity link for $m$). We experimented with many different classification models, including NB, LR, J48, RF, KNN, and SMO. We report results only for SMO which consistently had the best performance across datasets.

\subsection{Training and Labelling}
\label{subsec:training}

For training supervised classifiers on the prediction tasks, one can generate training instances using manual labelling (e.g., from domain experts) \cite{gerber2013real}, crowd-sourcing \cite{bontcheva2017crowdsourcing}, or existing ground truth datasets \cite{roder2017gerbil}. In our experiments, we make use of existing ground truth datasets (more in Sect. \ref{sec:evalSetup}). After annotating the documents of the training corpus, we compute the feature values for each mention that exists in the ground truth and assign the corresponding class labels.
For the \textit{multi-label classifier}, we label the training instances by simply considering the systems that managed to correctly disambiguate the mention.
For each \textit{binary classifier}, we make use of only the annotations produced by the corresponding EL tool and label the training instances as either \textit{true} or \textit{false}.

\section{Problem Definition}
\label{sec:problem}

We first define the notion of \textit{entity linking}:

\begin{definition}[Entity Linking]
Given a corpus of documents $D$, e.g., a set of news articles,
and a KB $K$, e.g., Wikipedia, that describes information for a set of entities $E$ (like persons, locations, organisations, etc.), the task of \textit{Entity Linking (EL)} aims at providing a set of \textit{entity annotations} $A$ of the form  $\langle m, e\rangle$, where $m$ is an entity mention recognised in a document of $D$ and $e$ is an entity in $E$ that determines the identity of $m$. An entity mention $m$ is of the form $\langle d, s, p\rangle$ where $d$ is a document in $D$, $s$ is a surface form in $d$ (a word or a sequence of words representing an entity), and $p$ is the position of $s$ in $d$. Thus, an entity mention is always identified by a document, a surface form, and a position. We denote by $M$ the set of all recognised entity mentions $\langle d, s, p\rangle$.
\end{definition}

Our approach exploits a set of $n$ EL systems $(l_1, .. , l_n)$ which operate on the same reference KB $K$ and are applied to the same corpus $D$.
The output is $n$ sets of entity annotations $\mathcal{A} = (A_1, \dots, A_n)$, corresponding to $n$ sets of entity mentions $\mathcal{M} = (M_1, \dots, M_n)$, each one produced by a different EL system $l_i$. The size of each set of entity mentions $M_i$ can be different, since each system might have recognised different entity mentions.

\begin{definition}[Meta Entity Linking]
Assuming that, for a given corpus $D$, we have $n$ sets of entity mentions $\mathcal{M} = (M_1, \dots, M_n)$ and $n$ sets of corresponding entity annotations $\mathcal{A} = (A_1, \dots, A_n)$, produced by $n$ different EL systems $(l_1, .. , l_n)$, the task of \textit{Meta Entity Linking}, for short \textit{MetaEL}, aims at providing a unified set of entity annotations $A_u$, where for each entity mention $m \in (M_1 \cup ... \cup M_n)$, the most correct annotation is selected from $(A_1 \cup ... \cup A_n)$.
\end{definition}

The solution proposed in this paper, called {\em MetaEL+}, is based on {\em supervised classification}. We propose two variations, one focusing on high recall (\textsc{LOOSE}) and a more selective one focusing on high precision (\textsc{STRICT}). In both approaches, if at least two of the considered EL systems have recognised and disambiguated the same entity mention $m$, we predict the system to consider using a \textit{multi-label classifier}.
If only one EL system has recognised an entity mention, the \textsc{STRICT} approach predicts if the provided entity link is correct using a system-specific \textit{binary classifier}. On the contrary, the \textsc{LOOSE} approach includes all annotations recognised by only one of the systems, thus focusing on high recall.

\section{Related Work}
\label{sec:rw}

The survey in \cite{shen2015entity} presents a thorough overview of the main approaches to EL, while more recent works (like \cite{cao2018neural}, \cite{kolitsas2018end} and \cite{fang2019joint}) exploit the idea of neural networks and deep learning.
To the best of our knowledge, \cite{ruiz2015combining}, \cite{corcoglioniti2016microneel} and \cite{canale2018novel} are the only previous works that focus on the related (yet different) problem of \textit{MetaEL}, i.e., on how to combine the outputs of multiple EL tools for providing a unified set of entity annotations.

\cite{ruiz2015combining} proposes a weighted voting scheme inspired by the ROVER method \cite{fiscus1997post}. This method ranks the candidate entities by considering the performance of the systems on a so-called ranking corpus. Two of our baselines consider this method.
\cite{corcoglioniti2016microneel} focuses on microposts and resolve conflicts by majority vote or, in the event of a tie, by giving different priorities to the annotations produced by each annotator. Two of our baselines consider this approach.
\cite{canale2018novel} describes a framework to combine the responses of multiple EL tools which relies on the joint training of two deep neural models. However, this work is not applicable in our MetaEL problem since it makes use of external knowledge (pre-trained word embeddings and entity abstracts) as well as entity type information (a type taxonomy from each extractor), as opposed to  our \textit{MetaEL} task which only considers plain lists of entity annotations.

With respect to the related problem of \textit{named-entity recognition} (NER), i.e., the detection of named entities in a given text and their classification in predefined categories like Person or Location, several works investigate how to combine the results of multiple NER methods \cite{dlugolinsky2014combining,corcoglioniti2016microneel,si2005boosting,plu2016enhancing}.
\cite{dlugolinsky2014combining} tackles the problem of concept extraction in \textit{microposts} and proposes machine learning methods that make use of features describing the microposts for combining the results of different NER tools. \cite{corcoglioniti2016microneel} also focuses on microposts and trains a multi-class SVM classifier.
\cite{si2005boosting} focuses on \textit{bio-medicine} and proposes three methods for combining the results of various bio-medical NER systems: i) majority vote, ii) unstructured exponential model that considers the performance of the systems on training data, and iii) conditional random field that models the correlation between biomedical entities. We use the first two methods as baselines in our experiments.
Finally, \cite{plu2016enhancing} unifies the outputs of three different named-entity extraction models (dictionary, POS tagger, NER) in a specific order and merges the overlapping mentions.

A related line of research on the NER problem combines multiple classifiers through ensemble learning \cite{wu2003stacked,florian2003named,saha2013combining,speck2014ensemble}.
\cite{wu2003stacked} examined several stacking and voting (majority-based) methods that combine three different classifiers.
In a similar way, \cite{florian2003named} combines the results of four classifiers,
while \cite{saha2013combining} constructs an ensemble of seven classifiers.
\cite{speck2014ensemble} evaluates the performance of 15 classification models, finding that ensemble learning can highly reduce the error rate of state-of-the-art NER systems.
These works use as features the \textit{predictions} of multiple supervised classifiers for deciding on the entity \textit{type} of a given mention (from a pre-defined list of entity types), as opposed to our MetaEL task which combines \textit{EL systems} and considers features extracted from the underlying \textit{corpus} for training dedicated classifiers able to predict the \textit{EL system} that can provide the correct link for a given mention.

A related interesting work is the NERD framework \cite{rizzo2011nerd} which allows running multiple EL systems on the same text(s). NERD uses a common ontology for storing the results, thus providing a common representation format and facilitating the evaluation of NER and EL methods. However, it does not resolve conflicts like in the case of MetaEL. Our work can be used by this framework for conflict resolution and for providing a single set of entity annotations.

redpajama_set_name: RedPajamaArXiv


Was Mary Surratt a Lincoln Conspirator?
Scream's iconic villain Ghostface was inspired by a real serial killer who preyed on young students in the college town of Gainesville, Florida.
When a high-tech helicopter like the Sikorsky S-92 needs to be transported on a cargo plane, it's crucial that every rotor blade is dismantled and stowed away with care. That's because each one costs over $200,000.


This is not actually a story about sports, though it was produced by Tom Rinaldi for ESPN.
It is as unusual to air a nearly 14-minute piece on Sports Center as it would be for anyone on your station to air a 5-minute piece.
But ESPN is smart enough to realize that it is the story that dictates the time limit, not the other way around.
Are there stories in your town waiting to be noticed, waiting to be told?


Stephen Yeboah's life trajectory reflects exactly what he is doing today. He lived and grew up in a typical mining community, Obuasi, which hosted for more than two decades Ghana's oldest mining operation. This is the motivation behind his core interest in commodities in Africa.
Stephen Yeboah is the co-founder of Commodity Monitor, a tech-based business that uses data analytics to generate exchange of information on sustainable production, supply and consumption of commodities in Africa, including mining, oil and gas. Commodity Monitor undertakes political and economic risks and opportunities analyses for private investors and governments.
He has more than eight years of experience in research and policy analysis in Africa's extractive sector, agriculture, energy, climate and was a Research Fellow at the Africa Progress Panel (APP), a non-profit chaired by the late former UN Secretary General Kofi Annan, from September 2013 to July 2016. As part of the work for the APP, he engaged in policy research, communications and stakeholder liaison at the highest level towards driving Africa's agenda in agriculture, energy, climate, natural resources and sustainable development in the global context.
Stephen has also worked as a research consultant at the African Natural Resources Center of the African Development Bank (AfDB), where he assisted in projects like the value chain analyses and the use of financial modelling in Africa's extractive sector. He has consulted for the Communications and External Relations Department of the African Development Bank as a Digital Development Communications expert, where he leads policy-oriented communications, including drafting Op-Eds and policy-related articles.
He has published more than 100 articles, blogs and papers on energy and climate, finance, agriculture, natural resources governance and sustainable development in Africa and holds a Master in Development Studies from the Graduate Institute of International and Development Studies (IHEID), Geneva, Switzerland. He had his undergraduate education at the Kwame Nkrumah University of Science and Technology.
He is currently undertaking a doctoral study at the Swiss Graduate School of Public Administration (IDHEAP), University of Lausanne, Switzerland, researching on the politics, institutions of mining and the outcomes on Ghana's development.


text:
Q: Mascaramento para campo data Quem já usou o type="date" do HTML5 certamente notou que, apesar da data estar visível no formato DD/MM/YYYY, quando você recupera o value pelo JavaScript, ela está no formato YYYY-MM-DD.
O que eu quero é justamente reproduzir esse mesmo efeito. O usuário vê a informação no formato X, escreve no formato X, mas na verdade está no formato Y.
Alguém saberia me dizer como é possível fazer esse tipo de mascaramento (se é que é possível)? Ou pelo menos aonde eu posso encontrar algum material relevante a minha duvida?
Eu estou procurando em outros fóruns e no lord Google, mas tudo o que eu encontro são as formatações costumeiras de como passar de um formato para o outro.
Sei que tem a turma do "para que fazer se já tem pronto", mas a minha proposta é aprender. Eu gostei do efeito criado e, se for possível reproduzi-lo, eu gostaria de saber como.

-- EDIT --
O efeito que eu estou tentando criar é esse aqui:

A: Bem, no HTML5 desde que o elemento tenha uma ID, esse elemento será armazenado em uma variável cujo identificado é a ID que foi definida no HTML. Então dentro de uma propriedade desse input você pode definir o texto que você quiser a partir de um evento.
No exemplo abaixo eu fiz assim: toda vez que o usuário digitar uma letra, ele pega o conteúdo da textbox, faz o tratamento que eu quero, e adiciona o texto tratado na propriedade hiddenValue da própria textbox, depois quando me for conveniente, eu recupero o valor desse modo.
<!DOCTYPE html>

<html>
    <head>
        <title>HTML Element as var</title>
        <meta charset="utf-8" />
    </head>

    <body>
        <input id="username" value="" placeholder="hamboldt" onkeyup="getText(this);"/>
        <button onclick="alert(username.hiddenValue);">Ver valor escondido.</button>
    </body>

    <script type="text/javascript">
        function getText(input) {
            input.hiddenValue = input.value + "hidden";
        }
    </script>

</html>

redpajama_set_name: RedPajamaStackExchange


text:
Tell me a story, or hear a story or read a story…
Convener(s): Jen Lunn
Participants: Robert Wells, Aliki, Dodger, Lucy Avery
Summary of discussion, conclusions and/or recommendations:
Lucy asked to be told a story about an astronaut.
Once upon a time there was an astronaut who was longing for trifle. He'd been in space for ages and one day came upon a hitherto undiscovered planet which looked barren from the window of the spaceship but when he stepped out into it he recognized it to be Basildon.
He had only visited Basildon once; as a child taken to visit his Aunt Eve. While visitingher they went for a walk and he had been baffled by the fact that she had a walking stick but didn't seem to need it as she swung it about most of the time.
Aunt Eve had no children and was so excited to have a child come for tea that she had made sugar sandwiches and trifle.
The grown up astronaut found himself walking down a street in Basildon and found the house of his Aunt Eve. He walked inside and it was empty but on the table was a plate of sugar sandwiches and a trifle. He ate them all (tricky since his spaceman gloves were pretty massive and slightly unwieldy!).
He then found a piece of paper and wrote her a note. Again the spacemean gloves made it pretty tricky to write so the handwriting was that of a small boy.
Dear Aunt Eve,
Thank you for the sugar sandwiches and the trifle.
Love from Ben
As the astronaut looked at the note he had just written he realized it was the thank you note his parents had asked him to write after that visit when he was a child.
The astronaut walked out of the house and headed back to where he'd left his spaceship. It wasn't there. And suddenly his spacesuit disappeared and he was a 7 year old boy again.
He was in heaven.
Lucy drew me a picture of my story.
Robert asked me for 3 words to inspire his story. I gave him Rabbit, Mountain, pritstick… Someone else added Abbatoir. This is his story…
Rabbit Mountain Pritstick Abattoir by Robert Wells
Once upon a time in a magical faraway land, for that was the right time and the right place, there was a pen full of bunnies. Sometimes they would multiply until the pen was straining with the sheer number of them, and then the gates would open and many of them would be led out, never to return.
On one particular bright and shiny morning, a young fox approached the pen and whispered to a particularly young, particularly small bunny "Hey, little bunny".
"My name's Reg" replied Reg, the small bunny.
"Well," said the Fox "Reg, do you know where the bunnies that leave the pen go? No, of course you don't. They go to an abattoir. That's a big scary building where big scary machines chop off their little bunny heads. Then their innards are sucked out and their bodies are baked until they shrink and they are filled with milk chocolate. When the chocolate sets, their bodies are broken away and what is left is a chocolate Easter bunny. Thousands of children eat them, never knowing the horrible conditions that the rabbits go through. Have a nice day, little Reg."
Well, Reg set about telling all the other bunnies. Well, trying to. They thought that he was mad. One night he dug a tunnel under the fence. He tried one last time to get some other rabbits to join him, but they wouldn't listen, and even if they wanted to they wouldn't have been able to fit in the tiny tunnel he had dug.
Reg escaped. He ran and he ran (two legs, two legs, as bunnies have to run) and he could feel the horrible black mass of the abattoir behind him. He ran and he ran (two legs, two legs) until he came to the foot of a mountain. He longed to stop. He longed to go back. But each time he took a breath he felt the horror again. He ran and he ran (two legs, two legs) up the mountain to where the green gave in and the white took over. His little legs ached and his mind swam and the he saw flashing knives and whirling blades and he ran. He ran and he ran and he ran and he ran (two legs, two legs, two legs, two legs). He ran until his legs gave in and he fell into a dark cave, and finally he stopped to get his breath back.
When his night vision, which he never knew he had, asserted itself, he saw a dark figure in the corner. "I'll not be scared" he told himself "Be it beaver or bear, I'll face it". So he creeped across the cave and sneaked up to the figure and he saw a pair of boots. And in the boots were trousers. And on top of the trousers was a coat. But above the coat there was no face. Just a skull stared back at him. "Only a silly old skellington; nothing to fear there".
But Reg had realised that he could be brave, and no stinky old building was going to get the better of him. He looked through the skellington's pack and tried to find something to help him save all his friends. There was a climbing axe, but he couldn't lift it. There were ropes but he lacked the strength to even uncoil them. In the end the only thing he could find that he could carry was a pritstick. So he took that.
He pushed it out of the cave and into the sun. He climbed on the stick and surfed it down the snow. When he ran out of snow, he rolled it down the mountain, running on top. And very very soon, he was back at the pen where he climbed back through his tiny hole, dragging the glue behind him.
A few days passed and Reg made his plans. When he saw that the pen was full of bunnies he opened the pritstick and rubbed it all over his fur. He coated every inch of his tiny fluffy body with glue until his fur was heavy and matted and he could hardly move. He worked through the night and in the by the dawn, the pritstick was empty and Reg was covered.
When the pen opened Reg made sure he was at the front of the crowd. He starred at the blades threatening his little neck. Nearer and nearer they came, but Reg didn't flinch. As the whirling knives of doom touched little Reg's fur there was a horrible grinding noise. The glue instantly stuck to the steel and clogged up the fine mechanisms. There was a crunching of gears and a whining of motors and it is at this point where the rabbits suddenly realised that this was not a good place to be. They tumbled over each other to get out and poured through windows and doors into the fresh air. Finally, the exhausted Reg dashed out as cogs pinged and zinged off motors behind him.
In the meanwhilst, the farmer who owned this bunny farm woke to a horrid rumbling. Davis the farmer dashed to the door of the abattoir. Inside the machines had failed entirely, but they waited until Davis was in the middle of the building to explode, bringing the building crashing down.
And they all lived happily ever after, apart from Davis who died, and the children of the world who got one less chocolate treat for Easter. Reg could never clean the glue from his fur, but he rolled around in daisies and was forever after known as the floral hero.
Aliki told me this story…
The Masters Tools
Once upon a time there was a girl who lived with her mother and her younger brother at the edge of a forest. Their farm had hard stony ground and it was difficult to get enough to feed themselves. So, as soon as the brother was old enough to help work the farm, the girl set off to a market town on the other side of the forest to find work.
She was walking in the forest when she saw some kids stealing eggs from a birds nest and she stopped them. She talked them out of it and she went on her way.
She was walking along the river when she saw a man about to chop down an apple tree to build a house. She stopped him and talked him out of it. She explained that his children could play in the tree and pick apples. So he decided to build his house next to the tree instead.
As night was falling she met an old woman who asked her for some food and she shared the food her mother had sent with her from the farm. The old woman thanked her and gave her a crude clay bowl in return.
She got to the market town the next day and at the hiring fair she was hired to clean house for a magician who lived in a tall tower in the forest.
After she had worked for a month the magician refused to pay her and when she protested he locked her in the kitchen and told her she would have to sleep there. Every night the magician did terrible magic and the tower was filled with smells and horrible noises and every morning the girl would get up and clean the tower which was full of mysterious oozes and horrible stains.
And this went on for months and months and still the magician refused to pay her.
Finally, she decided that she was going to have to break out of the tower, take the money owed to her and return home. So she took the poker from the fire and tried to smash down the kitchen door but as she did the poker started to shout "Master! Master! Your servant is trying to escape!" And the magician came down to the kitchen, beat her and locked her back in the kitchen.
So the next night she tried to use a knife from the kitchen drawer to pick the lock but the knife started to shout "Master! Master! Your servant is trying to escape!" And the magician came down to the kitchen, beat her and locked her back in the kitchen.
The third night came and she didn't know what to do so she started to cry. She held the bowl that the old woman had given her as it was the last memory of kindness that she had.
Her tears filled up the bowl, poured over the edge and bagan to fill the cracks between the flagstones. As she was crying she heard birdsong. The bird whose nest she had saved was at the small barred window of the kitchen. It flew in and dropped from it's beak the seeds from an apple from the tree she had saved. The birds planted the seeds amongst the flagstones and her tears watered them.
The seeds started sprouting and they grew quickly. Exhausted she huddled at the base of what soon became a tree. And the trees kept on growing. Within a few hours the trees were so big they had destroyed the tower completely.
The magician tried to stop her but his magic had no power over the trees and he ran away. He was so scared that he left all his money behind. The girl plucked the bags of money from the branches of the trees and took them home to her family.
Dodger told me a true story about an experience he had had training with a guy from the Bejing Opera circus.
He spent a week or so training with this man and felt he was learning loads and having an amazing time. The teacher did not speak English and Dodger did not speak Mandarin. Over the time Dodger began to hear one phrase repeated over and over "Fa La" and grew to like the phrase.
His friend who was the translator for the teacher asked him at the end of the project how he had enjoyed it and how he had managed with the langusge barrier. Dodger explained that he had had an amazing time and was so excited to have learnt so much in such a short time. He said the language hadn't been a problem. He repeated the phrase "Fa La" to his friend saying that the teacher had said this all the time. What did it mean?
"Not like that."
Published: 19th April, 2018
Devoted & Disgruntled 7
Bridging the gap between core programming and education/participation work
How can we better develop the connection between main stage and education/participation work? Is there a stigma attached to participation work from an audiences perspective that means they don't attend performances with this at its heart? Do audiences fear it will be of lesser quality therefore not want to risk wasting their valuable money?
Bel Canto: Cog or Creative?
A wide-ranging discussion about the role of the Vocalist within Opera
"REGIETHEATER" (Or, what are we going to do about Directors?)
We discussed the concept of "Regietheater" - what it means. Some people had heard the word 'Eurotrash' also applied to this type of opera production which seemed to provoke some strong reactions.

redpajama_set_name: RedPajamaCommonCrawl


Convert your Wi-Fi into a state-of-the-art marketing tool. Build your customers database. Promote your Facebook or Twitter Fan Page, gain more fans, followers, check-ins and tweets.
No complicated passwords to share, just quick and simple Social WiFi guest access. Customers get online using a social media account or email address.
Customers connect to your SSID Wi-Fi Network.
Customers login to your WiFi through their social media accounts such as Facebook, Twitter, Google+.
Simple promote your Facebook fan page, gain more fans & check-ins and/or promote your Twitter page, gain more followers & tweets.
Turn your Free Wi-Fi service into an automated lead generating system. Engage your customers on Social Media and gain valuable marketing data. Run contests, promotions and special offers, engaging your customers in real-time.
Proven solution used by satisfied customers from Europe, USA, Canada, South America, Asia, Africa and Australia as well.
SOCIFI is proud to be compatible with the best & reliable vendors such as AntLabs, Aruba, Cisco, Cisco Meraki, DD-WRT, EnGenius Europe, Icomera, LigoWave, Mikrotik, Open-Mesh, Ruckus, Xirrus, ZyXEL and many others.
Upgrade, downgrade or cancel anytime. No commitments.


text:
Register >>
About Jerusalem
Covid Regulations
רישום לנרשמים מקומיים
Instructions for Oral Presenters
Poster Instructions
OPTIONAL PRE- AND POST-TOURS
Pre-Tour A
Click here to book your optional tours
JERUSALEM & BETHLEHEM
Friday, 16 September, 2022
US$130 per person
09:00 Meet in the lobby of the Notre Dame of Jerusalem hotel. Go up to the roof top for a quick explanation before driving out to the 'New City' where you will start the day at the Israel Museum. Enjoy a guided tour of the Model of the Holy Land from the second temple and the Dead Sea Scrolls. Continue to Bethlehem – the birthplace of Jesus and visit the Church of Nativity and view the Shepherd's field. Return to Jerusalem to visit the Old City. The tour of the Old City will entail a lot of walking and will include the Western Wall, the Christian Quarter with some of the Stations of the Via Dolorosa and culminate at the Church of Holy Sepulchre. Enjoy free time to roam around the Arab Market before returning to your hotels on foot.
Pre-Tour B
JAFFA, TEL AVIV & GRAFFITI TOUR
Start the tour with a visit to Jaffa – the ancient port city of Tel Aviv and one of the oldest inhabited places in the world. On to Tel Aviv to walk the famous Rothschild Boulevard with a guided description on the Unesco World Heritage Bauhaus Architecture.
Continue to Florentin neighborhood of South Tel Aviv for a graffiti tour, where you will learn about this amazing neighborhood through its history, culture and especially its beautiful street art, all of which helps us better understand modern Israeli society.
Experience the Tel Aviv 'shuk', followed by a visit to Nachalat Binyamin, a local Israeli artists' market, before returning to Jerusalem.
Pick up at 08:00 and drop off approximately 18:00.
Pre-Tour C
TOUR CANCELLED
Saturday, 17 September, 2022
Pre-Tour D
Post-Tour E
SOREQ CAVE & WINERY
09:00 Meet in the lobby of the Notre Dame hotel. Visit the roof top for a quick explanation. Continue to the incredible Soreq Cave – one of the most beautiful stalactite caves in the world. Following your visit 'underground' come out to the 'light' and enjoy some wine in the hills of Jerusalem. End the day with a visit to Ein Karem, known as the birthplace of John the Baptist, and take in the beauty of this picturesque village. Drop off will be at the three official hotels: Dan Panorama, Mamilla and Notre Dame at approximately 17:00
Pre-Tour E
DEAD SEA & MASADA
Depart Jerusalem for a drive via the Judean Desert to the Dead Sea area. Take the cable car up to Masada for a tour of the remains of the 2000-year-old fortress and palace built by King Herod. Following the tour, experience a float in the Dead Sea - 400 meters below sea level, the lowest place on earth.
Note: Sun hat, walking shoes, swimwear, sunscreen and swimming shoes are recommended. A towel and locker will be provided for your convenience.
Post-Tour F
EILAT & PETRA
Wednesday-Friday, 21-23 September , 2022
US$ 1,550 per person sharing in a double room
US$ 420 single supplement
Wednesday, September 21:
After breakfast, check-out of the hotel and head south driving through the Negev. Arrive at Mitzpe Ramon and take in this amazing site - one of the geographical wonders. The guide will give you an overview of your surroundings. Visit the visitors center before enjoying a Bedouin lunch experience learning about their lifestyle and experience a camel ride. Continue to Eilat for overnight at the Dan Panorama hotel [or similar].
Thursday, September 22:
Meet our representative in the lobby of the hotel and transfer to the Arava border. Walk 200 meters towards the Aqaba border, where your English-speaking Jordanian guide will be waiting for you. Depart for Petra and visit this incredible Nabatean city, which was lost for hundreds of years and found some 200 years ago. Lunch in Petra, before returning to the border. Overnight in Eilat.
Friday, September 23:
Transfer to the local airport for your flight to Tel Aviv. Overnight at the Lighthouse hotel [or similar].
All tours include:
All tours are based on minimum 15 participants
English speaking licensed guide
Transport in airconditioned vehicles
Entrance fees as per itinerary
Lunch/drinks/incidentals
Tips for the guide and the driver
In order to book optional tours kindly complete the online accommodation form.
For further accommodations and/or touring please contact the organizers:
Target Conferences Ltd.
Email: tmh.reg@target-conferences.com
Tel: +972 3 5175150, Fax: +972 3 5175155
Payment for tours reserved directly by Target Conferences is to be made at the time of booking. Payment may be made by bank transfer* or credit card (Visa, Mastercard, American Express, Diners)
Bank transfers should be made to the following account:
Bank Leumi
Ahad Ha'am Branch number 811
31 Montefiore St. Tel Aviv
Account name: Target Conferences Ltd.
Swift: Lumiilit
Account number: 610500/25
IBAN: IL050108110000061050025
*Please ensure that the name/s of the participants and conference name (TMH) are clearly stated on the bank transfer. Bank transfer charges are the responsibility of the payer and should be paid at source in additional to the sum being transferred.
Our refund policy on tours is based upon receiving notification in writing as follows:
Until 14 August, 2022 - Full refund less handling fee of $50
As of 15 August, 2022 - Officially no refund, on a case-by-case basis
BACK TO TOP >
T: +972-3-517-5150
F: +972-3-517-5155
The Secretariat will be happy to assist you with any additional information
General Inquiries / tmh@target-conferences.com
Registration & Accommodation / tmh.reg@target-conferences.com
© Target Conferences 2020. All rights reserved. This site is best viewed using Internet Explorer (version 9 and above), Firefox, Chrome and Safari


The Escorpión at Jettison is a smoke show and will definitely impress a date.
It feels like date night keeps getting more and more expensive.
The prices at hot new restaurant openings are trending ever upward. As diners look for innovative, interesting concepts that don't thin out our wallets, industry investors seem more excited to invest in high-concept eateries that cater to the comfortably well off.
Case in point: upcoming French bistro Bullion. Owner Bruno Davaillon told the The Dallas Morning News, "I want a place where people can come two or three times a week." He also said, in the same interview, that meals will be about $80 per person.
Most of us can't spend $80 on dinner three times a week. But we certainly want to enjoy the occasional classy date night. An intimate environment, something interesting to drink, damn good food: Surely that's still affordable somewhere, right?
Here are seven restaurants — and one bar/restaurant crawl — we wholeheartedly recommend as elegant, enjoyable and absolutely date-night worthy without blowing your budget. We used a baseline of about $50 for food for two people, although many of the suggestions below are so affordable that $50 to $60 for two will also cover drinks.
Time to text your partner or start swiping right a little harder: Classy yet affordable is back.
Midcentury modern aesthetic is strong at Sassetta.
Sassetta's a little loud, but it's also a rare combination: exquisitely trendy and surprisingly reasonable. Push through the comically huge front door to a sunlit interior full of warm colors and white-clad waiters proffering fresh pizzas. A shared salad, pizza (with a perfectly snappy crust and, after an appetizer, just the right size for two) and a glass of wine per diner will come out to just about $50, but it's worth spending a little extra on a slice of the lemon cake topped with pine nuts or on an extra glass from Sassetta's extensive, adventurous, all-Italian wine list.
Sixty Vines brings elegant and modern but affordable date-night vibes to the 'burbs.
Sixty Vines is another riff on the thin-crust, fresh-veggie pizza trend, with a surprisingly decent house label of California wines and an extensive list of vino on tap. (Yes, tap.) Plus, it's in Plano, making this the perfect place for workers around Legacy to enjoy happy hour or a bowl of fettuccine with diced squash and a sunny side up egg yolk perched on top. One caveat: Sixty Vines is loud, popular and often very busy. What it lacks in intimacy, the huge restaurant more than makes up for in first-date conversation points. Oh, and a couple of bowls of pasta, a couple of glasses of wine and a portion of devil's food cake with salted-caramel ice cream will come out to about $50.
Lima Taverna's pescado a lo macho is a big, boisterous parade of seafood, almost all of it perfectly cooked.
One of the most exciting Plano openings of 2017, Lima Taverna plates fabulous seafood, from the pescado a lo macho stew to the often-fiery ceviches. Its interior is a nostalgia-filled trip to Peru, and the excellent pisco sours served at the bar certainly don't hurt. A Peruvian dinner might be a good way to find out if your date has adventurous tastes, although if not, the restaurant serves a first-rate grilled pork chop, too. Either way, you will look in-the-know for recommending an elegant restaurant that still qualifies as an underappreciated hidden gem.
Lanzhou Dan-Dan Laiman noodles, hand-pulled, for $11.
Ask, if you can, to sit at the noodle bar, where some of Dallas' best dough experts are pulling fresh noodles by hand and forming dumplings from scratch. Order soup dumplings with paper-thin wrappers, made before your eyes, containing pork and a scrumptious savory broth, a sort of culinary magic trick. Don't miss dan-dan noodles, either, in any of the varieties. There's other food on Royal China's menu, too, and it's pretty consistent, but once you see the noodle makers at work, it's not likely you'll feel a desire to order anything else.
A sweet apple swan caps off a meal at Hon Sushi.
Apples carved into wide-winged swans. Sushi rolls presented in pastoral landscapes under palm trees carved out of cucumbers. Wasabi molded into the shape of a hot pepper. Blocks of ice with parsley sprigs frozen inside. The sushi chefs at Hon Sushi are true artists, with a flair for the visual that matches their honest craftsmanship. In downtown Dallas, Hon Sushi might be prohibitively expensive and filled with suits, but it's improbably housed in a Carrollton strip mall, where the foodie masses still haven't found it. I can vouch for this one: I went on a second date here 14 months ago (for this review), and she hasn't dumped me yet.
The Grape's martini has olives stuffed with Boursin cheese.
The Grape has been a Dallas legend — and a first choice for date nights — for 45 years. The restaurant is still getting it right, with impeccable service and seasonally rotating menus. Still, as much as the menu changes with the harvest, at least one friend of mine is addicted to the mushroom soup and gets a $4 cup every time. The Sunday brunch is peerless without being pricey, especially if your date doesn't mind burger grease running down your fingers. May we also recommend a special-occasion splurge on the monthly "Come As You Are" dinner, with three specially chosen courses and wine pairings for $54 per person?
Arrive at Boulevardier before 6:30 p.m. on a Friday, and the good bottles of bubbly are 50 percent off.
Here is an essential bit of knowledge: Happy hour at Boulevardier means discounted oysters. Here is another essential bit of knowledge: Arrive at Boulevardier before 6:30 p.m. on a Friday, and the good bottles of bubbly are 50 percent off. In other words, if you and your date find yourselves leaving work early on a Friday, there's no better place to go. Oysters are allegedly an aphrodisiac, if you believe in that sort of thing. If you don't, well, Champagne gets you drunk.
The Sylvan Thirty development has done a fantastic job building a little date crawl. Start with a light bite for dinner at Whisk Crepes, the cozy Parisian café that somehow escaped and moved to Dallas, or CiboDivino, the Italian market with delightful pasta and pizza. At Whisk, split a big bottle of traditional French cider; at CiboDivino, opt for craft beers or Italian wine. But don't party too hard because you've still got to stop by one of the city's most sophisticated cocktail bars, Jettison, where the refinement of the drinks is matched only by the friendliness of the staff. If you can ever bring yourself to leave, swing by new Betty Ringer for a scoop of an ice cream flavor like Oaxacan chocolate, horchata or roasted banana pudding.


text:
Home | Legal Advice | Judgment | Ambalal Babulal Patel Etc. Versus The Group General Manager, ONGC & Another [2022]
Ambalal Babulal Patel Etc. Versus The Group General Manager, ONGC & Another [2022]
Ambalal Babulal Patel Etc. Versus The Group General Manager, Ongc & Anr. [2022] Civil Appeal No(S). 385386 Of 2022 (Arising Out Of Slp(Civil) No(S).70867087 Of 2017)
1. Leave granted
2. The appellants are the claimants and being dissatisfied with the amount of compensation awarded by the Division Bench of the High Court under the impugned judgment, have approached this Court by filing these instant appeals.
3. The subject lands in question are situated at Villages Pansar, Dhamasana and Isand which came to be acquired under the provisions of the Land Acquisition Act, 1894 (hereinafter being referred to as the "Act") for the public purpose for ONGC. The notification under Section 4 of the Act came to be published and after going through the procedure prescribed under law, the Special Land Acquisition Officer awarded compensation under Section 11 of the Act.
4. Upon reference being made by the appellantsclaimants in respect of the lands acquired at Villages Pansar/Dhamasana/Isand, the Reference Court awarded additional compensation, over and above the compensation awarded by the Special Land Acquisition Officer, ONGC, Ahmedabad, under its respective orders along with interest and other statutory benefits under Section 23(1)(A) of the Act.
5. The orders passed by the Reference Court came to be challenged by respondent no.1 (ONGC) before the High Court and after hearing the parties, the High Court under its respective impugned judgments, while upholding the statutory benefits including interest, modified the rate of compensation and reduced the additional compensation for the land acquired at Villages Pansar/Dhamasana/Isand, Taluka Kalol, District Gandhinagar, which is the subject matter of challenge at the instance of the appellantsclaimants in the present batch of appeals.
6. After we have heard learned counsel for the parties, in our considered view, the Reference Court in exercise of its power under Section 18 of the Act, after appreciating the material available on record, awarded additional compensation to the claimants over and above the compensation awarded by the Special Land Acquisition Officer, ONGC, Ahmedabad and the interference made by the High Court under the impugned judgment so far as the additional compensation assessed by the Reference Court is concerned, is neither supported by the material on record nor sustainable in law.
7. Consequently, additional compensation awarded by the Reference Court in the respective orders stands restored. The impugned judgment of the High Court, to the limited extent is accordingly modified.
8. The appeals in the above terms stand disposed of.
9. Pending application(s), if any, shall also stand disposed of.


Executive Vice Foreign Minister Zhang Yesui and First Vice Foreign Minister of the Republic of Korea (ROK) Lim Sung Nam will hold the eighth high-level strategic dialogue between the foreign ministries of China and the ROK in Beijing on June 20.
Q: In the EU today, Greece blocked a planned statement by the EU that would condemn China's human rights record. Has China communicated with Greece on this issue or the EU more broadly to block this statement? Do you have any comment on Greece's decision to stop that statement?
A: The Chinese side has noted relevant reports. China's position on the issue of human rights is quite clear-cut. We maintain that all relevant parties should conduct constructive dialogue and exchanges on human rights and oppose politicizing the issue of human rights and interfering in the internal affairs and judicial sovereignty of other countries under the pretext of human rights.
The Chinese side stands ready to, on the basis of equality and mutual respect, strengthen exchanges and cooperation in the field of human rights with all relevant parties including the European Union (EU) to learn from each other, achieve common progress and make concerted efforts to contribute to the development of the cause of international human rights.
Regarding the discussions on this issue within the EU, I don't know relevant details. What I can tell you is that we applaud relevant EU country(ies) for sticking to the right position.
Q: What topics will be discussed at the upcoming eighth high-level strategic dialogue between the foreign ministries of China and the ROK? What are the expectations of the Chinese side on this dialogue?
A: At present, the China-ROK relations is in an important period. The two sides will take the opportunity of this dialogue to exchange views on China-ROK relations, the situation on the Korean Peninsula and other issues of mutual interest. The Chinese side hopes that this dialogue will play a positive role in strengthening bilateral communication, enhancing mutual trust, promoting the proper settlement of differences and improving bilateral relations.
Q: Does China has the impression that the BRICS can fill the void the US is leaving in areas like free trade or fight against global warming?
A: My colleague released the information on the Meeting of BRICS Ministers of Foreign Affairs last week.
The Meeting of BRICS Ministers of Foreign Affairs was held in Beijing today. During the meeting, Foreign Minister Wang Yi met with the press and answered questions. According to my information at hand, the meeting just concluded. The Chinese side will release the relevant information on the details of the meeting.
On your question about the role of BRICS countries in international affairs, they are all developing countries and emerging markets with important influence. The stronger exchange and cooperation between the five countries have important implications for upholding world peace and stability, boosting global economic growth, promoting democracy in international relations and improving global governance. The five countries have very strong will in these aspects.
Looking back at the past decade, BRICS cooperation has achieved very positive and prominent results. We will work with other BRICS countries to advance the development of BRICS cooperation in a more in-depth way in the next decade. The next BRICS Summit will be held in Xiamen in September. We stand ready to work with various parties for positive outcomes and a full success of this meeting.


There are plenty of ways on how to become an extra for a movie or film, and among these ways would be through movie extra auditions. Basically, the job of a movie extra is to fill in the background of any film when the camera starts to roll. Since it would be empty to have a set only with actors who are talking to each other, directors would prefer hiring movie extras to make everything appear to be natural. Below are some ways to get in and things to remember once you're on the set.