Transformer artificial neural networks have shown spectacular success in machine translation and answering search queries by internally extracting, after extensive pretraining, abstract patterns, and long-range dependencies in human language samples through encoder hierarchies \cite{vaswani2017attention}. Therefore, it seems natural to apply such schemes to other tasks that traditionally depended on human language understanding. Thus, this paper uses versions of BERT to investigate questions of authorship in Ancient Greek literature. Such models have already been used for authorship attribution, but the
actual number of cases is still limited. \citet{fabien2020bertaa} demonstrate that fine-tuning of a pretrained BERT  language model with an additional dense layer and a softmax activation allows performing authorship classification. \citet{polignano2020contextualized} use BERT for author profiling in social media and conclude that despite encouraging results in terms of reliability, the computational power required for running such a model is too demanding for the task. These results show that transformers could be successfully used for authorship attribution, yet the application of these models to historical texts is still limited. Specifically, \citet{bamman2020latin} develop BERT for Latin and \cite{assael2019restoring} train an LSTM language model of Ancient Greek. There is also a char-BERT implementation\footnote{https://github.com/brennannicholson/ancient-greek-char-bert}, but we do not know of any public full BERT model trained to work with Ancient Greek. We are also unaware of any examples when BERT was used successfully for authorship attribution of historical documents. 

\begin{itemize}
    \item we use a transfer learning approach to train BERT for Ancient Greek;
    \item we demonstrate that this language model is useful for authorship attribution of Ancient Greek texts;
    \item we obtain results that may be used as evidence in the process of authorship attribution of the Pseudo-Plutarchean texts;
    \item we obtain new insights into the paths along which the reception of ancient philosophy was developed.
\end{itemize}

\section{Data}
The data were taken from the digital history projects at the Chair of
Ancient History of the University of Leipzig. The Plutarch texts were transformed into a digital representation under professional supervision. Data can be obtained from the following free repositories Perseus Digital Library\footnote{https://github.com/PerseusDL/canonical-greekLit} and First Thousand Years of Greek\footnote{https://github.com/ThomasK81/TEItoCEX} as part of Open Greek and Latin\footnote{https://opengreekandlatin.org}. The resulting representation was stored in XML format (TEI guidelines) and enriched with metadata. The XML structure and metadata were removed. The strings were transferred into lowercase letters. The diacritics were removed. We did not touch hyphenation, punctuation, or any multilingual remains, nor did we apply any special language-related transformations.
The resulting data set consists of 1 244 documents with 199 809 paragraphs or 14 373 311 words.

\subsection{Tokenizers}
The tokenization of words into sub-word tokens is a crucial preprocessing step that can affect the performance of the model. Up to this point, we were using "words" as a linguistic term, whereas we understand tokens as the output of a tokenization algorithm. Thus, there would be one or more tokens that represent every word. Counting the number of tokens used on average to represent a word or, conversely, an average number of words per token gives an estimate of how fit  the tokenization is for the data set. In particular, the average number of words per token varies from 0 to 1, and the closer it is to 1, the more words are represented with one token. Various researchers have shown that corpus-specific tokenization could be beneficial for an NLP task. For example, \citet{sennrich-etal-2016-neural} show that optimal vocabulary is dependent on the frequencies of the words in the target corpus. \citet{lakew2019controlling} and \citet{aji2020neural} partially discuss the tokenization in the setting of cross-language transfer. Though \citet{aji2020neural} demonstrate that there is no clear evidence that one parent is better than another for cross-lingual transfer learning, they also show that token matching and joint vocabulary \cite{nguyen2017transfer} are the best ways to handle the embedding layer during transfer learning. 

Since each model has its own specific tokenizer before pretraining, one wants to measure how well each of them works with Ancient Greek. To do that, one could take two sample data sets: a sample of Modern Greek
Wikipedia (referred to in Table \ref{tab:tok} as “modern”) and a comparable sample of Ancient Greek (“ancient”  in Table \ref{tab:tok}). Each data sample is tokenized with both the Modern Greek BERT tokenizer\footnote{https://huggingface.co/nlpaueb/bert-base-greek-uncased-v} and the Multilingual BERT tokenizer\footnote{https://huggingface.co/bert-base-multilingual-cased}. One can speculate that the model uses shorter tokens to adopt grammatical information and deal with longer, rarely observed words. In contrast, the representations with longer tokens could be useful for semantically intensive problems. These longer, semantically charged tokens may vary significantly on various downstream tasks. Thus, the average length of a token and the average number of words per token, shown in Table \ref{tab:tok}, could be coarsely used as an estimate of the resulting tokenization. One could claim that the higher these values, the more apt the tokenizer is for the task. Indeed, higher average length of a token and number of words per token mean that longer, more semantically charged tokens could be matched for transfer; see \cite{singh2019bert,aji2020neural,samenko2021fine} for a detailed discussion of various tokenization properties.

\subsection{Training Ancient Greek BERT via Transfer Learning}
If related tasks are available, we can fine-tune the model first on a related task with more data before fine-tuning it on the target task, see \cite{ruder2019transfer}. This helps particularly for tasks with limited data \cite{phang2018sentence} and improves sample efficiency on the target task \cite{yogatama2019learning}. Since we have a limited amount of Ancient Greek texts, we want to do language transfer from Modern Greek to Ancient Greek training a masked language
model (MLM)\footnote{https://github.com/huggingface/transformers/blob/master/ examples/language-modeling/run\_language\_modeling.py} of the Ancient Greek text. 

We use this data set to train BERT Classifiers similarly to \cite{fabien2020bertaa}. Table \ref{tab:au} shows the validation accuracy for Modern Greek and Multilingual BERTs after MLM on Ancient Greek and 10 epochs of classifier training\footnote{AdamW, LR = 2e-5, eps = 1e-8,  linear\_schedule}. Table \ref{tab:au}  also shows a standard NLTK Naive Bayes Classifier trained on the 2 000 most frequent unigrams as a reference point for authorship attribution accuracy.

Since Ancient Greek was used in various regions and territories, one might expect that the texts also show regional peculiarities. Such peculiarities
then should also be detectable by a dedicated regional BERT Classifier. We constructed three coarse regions for the origins of the authors in the data set: a region surrounding Delphi, where Plutarch was working, a region in the proximity of Alexandria, and the region of ancient Ionia, namely the ancient region on the central part of the western coast of modern Anatolia, see Figure \ref{fig:map}. After balancing texts written by authors in these three regions with the fourth label that includes random sentences from authors outside of these regions, we train another BERT-based classifier to achieve a $0.79$ validation accuracy for the region of the author. Table \ref{tab:region} shows the results of the obtained classifier on the validation set.

\begin{table*}[h]
\centering
\begin{tabular}{lrrrr}
\hline
               \multicolumn{1}{c}{Predicted Region}      & \multicolumn{1}{c}{Pergamon Region} & \multicolumn{1}{c}{Alexandria Region} & \multicolumn{1}{c}{Delphi Region} & \multicolumn{1}{c}{Other Regions} \\ \hline
 Pergamon   & \textbf{83\%}                               & 3\%                                  & 3\%                             & 7\%                              \\
 Alexandria & 5\%                                & \textbf{77\%}                                 & 7 \%                             & 10\%                               \\
 Delphi      & 4\%                                & 5\%                                   & \textbf{81\%}                             & 8\%                              \\
 Other      & 8\%                                & 15\%                         & 9\%                             & \textbf{75\%}                             \\ \hline
\end{tabular}
\caption{Results of the BERT-based regional classifier on 4000 sentences set aside for validation.}
\label{tab:region}
\end{table*}

You are provided with a table and supplementary text from a machine learning paper. Here are JSON templates for four types of numeric cells: "Other", "Data Stat.", "Hyper-parameter/Architecture", and "Result":
{"value": "xx", "type": "Other"}
{"value": "xx", "type": "Data Stat.", "dataset": "xx", "attribute name": "xx", "sub-set/group name": "xx", "dataset features": {"xx": "yy"}}
{"value": "xx", "type": "Hyper-parameter/Architecture", "model": "xx", "parameter/architecture name": "xx", "dataset": "xx"}
{"value": "xx", "type": "Result", "task": "xx", "metric": "xx", "training data/set": "xx", "test data/set": "xx", "model": "xx", "model settings": {"xx": "yy"}, "experimental settings": {"xx": "yy"}}

Please describe all numeric cells in the above latex table following the JSON templates (proceeding by row in a left-right, top-down direction). For each cell, output one JSON description per line. For any unanswerable attributes in the templates, set their value to the placeholder "xx" if it is of string type and {"xx": "yy"} if it is of dictionary type. Numeric cells, which describe performance/error analysis, should be labeled as "Other".

Cell Description:
{"value": "83", "type": "Other"}
{"value": "3", "type": "Other"}
{"value": "3", "type": "Other"}
{"value": "7", "type": "Other"}
{"value": "5", "type": "Other"}
{"value": "77", "type": "Other"}
{"value": "7", "type": "Other"}
{"value": "10", "type": "Other"}
{"value": "4", "type": "Other"}
{"value": "5", "type": "Other"}
{"value": "81", "type": "Other"}
{"value": "8", "type": "Other"}
{"value": "8", "type": "Other"}
{"value": "15", "type": "Other"}
{"value": "9", "type": "Other"}
{"value": "75", "type": "Other"}