text:
\section{Introduction}
\label{SecIntro}

There is a lot of debate in various research communities on whether to anonymize author identities or not. The primary argument for anonymization is that it mitigates bias in the review process. Indeed, a number of experiments in computer science~\cite{tomkins2017reviewer,manzoor2020uncovering} and in other fields~\cite{okike2016single,blank1991effects,ross2006effect,garfunkel1994effect,fisher1994effects} have found evidence that reviews are biased if author identities are visible to the reviewers. The most consistent finding across these studies is that reviews are biased favorably towards authors from higher-ranked affiliations or authors who are famous.

Traditionally, peer review in the field of theoretical computer science does \emph{not} anonymize author identities. This topic has been hotly debated in the field with various opinions and discussions~\cite{Fortnow2018,Barak_2018,Venkatasubramanian_2018,Reingold_2018,Mitzenmacher_2018}. Some of the stated reasons against anonymization pertain to using author identities to gain more confidence in hard-to-very mathematical proofs or wacky ideas, and allocate reviewers' limited time and effort accordingly.

The review process of the ITCS 2023 conference thus took a middle-ground approach. Author identities were initially hidden from reviewers. Once the reviewer submitted their review, they could see author identities. Reviewers were subsequently allowed to modify their review if they wished. The rationale behind this approach was that it could mitigate bias at least in the initial impression, and later allow reviewers to use author identities in a manner that has been posited in aforementioned discussions. Note that the authors were free to upload their (non anonymous) papers to preprint servers, social media, etc.

In this paper we address three key questions pertaining to author identities in the review process:
\begin{itemize}
\item \emph{Identification:} Given that author identities were initially hidden, how many reviewers could identify the authors?
\begin{itemize}
    \item We address this question by asking reviewers to self-report their knowledge of author identities when they submit their review (Section~\ref{SecBasicReview}).
\end{itemize}
\item \emph{Usage:} Given that author identities were made visible to reviewers after they submitted initial reviews, how did the reviewers subsequently use the author identities to modify their reviews?
\begin{itemize}
    \item We address this question by analyzing the change in the reviews after they are initially submitted (Section~\ref{SecChangeAnalysis}).
\end{itemize}
\item \emph{Opinions:} What opinions or prespectives do participants have regarding anonymization of author identities?
\begin{itemize}
    \item We address this question via an anonymous survey of the participants in the ITCS 2023 review process (Section~\ref{SecSurvey}).
\end{itemize}
\end{itemize}
We envisage these results to be useful in designing author-anonymization policies in an evidence-based fashion.

\section{Related work}

A number of studies investigate whether revealing identities of authors to reviewers results in biases in the reviews. Perhaps the most well-known such experiment in computer science involves a randomized controlled trial at the WSDM 2017 conference~\cite{tomkins2017reviewer} in data mining. This experiment found strong biases favoring authors who were famous or from top-ranked institutions. (See~\cite{stelmakh2019testing} for some concerns regarding the experimental design and methods employed in~\cite{tomkins2017reviewer}.) The study~\cite{manzoor2020uncovering} conducted an observational study that found biases favoring authors from top-ranked affiliations in the ICLR conference in machine learning. A number of studies in other fields also find biases favoring authors who are famous or from top-ranked institutions~\cite{okike2016single,blank1991effects,ross2006effect,garfunkel1994effect,fisher1994effects}.

The studies~\cite{ware2008peer,mulligan2013peer} survey researchers about their perceptions of the peer-review process. The surveys reveal that researchers across a number of fields prefer anonymizing author identities. In the adjoining field of information theory where anonymization is currently not the norm, a survey of the members of the community~\cite{shah2021itsocsurvey} shows significant support for trying out anonymization.

The problem of measuring possible bias in peer review has also spurred research on designing and evaluating the methods to do so. The aforementioned work~\cite{stelmakh2019testing} provides an experimental design framework, statistical tests, and associated theoretical guarantees for conducting such a measurement in a randomized controlled trial. The paper~\cite{manzoor2020uncovering} uses difference-in-difference techniques to estimate bias exploiting the fact that the ICLR conference changed from not anonymizing to anonymizing author names in a certain year. In addition, the paper~\cite{manzoor2020uncovering} also develops methods to test for biases based on the review text rather than just the review scores. The pair of papers~\cite{madden2006impact,snodgrass2006single} debate the methods of computing the bias, with different metrics leading to different conclusions. Finally, the paper~\cite{jecmen2021split} focuses on the tradeoff between running such a controlled experiment in the peer-review process and the efficiency of the peer-review process.

Even when author identities are not included in the submissions, there are ways in which they may be deciphered. First, a reviewer may have seen the paper outside of the review process such as on a preprint server, on social media, or in a talk. For instance, a little over half of the papers submitted to the NeurIPS 2019 conference were posted on arXiv, and among these papers, 21\% were seen by at least one reviewer~\cite{beygelzimer2019neurips}. Second, some reviewers may actively search for their assigned papers online. For instance, the study~\cite{rastogi2022arxiv} found that over a third of the reviewers in two top conferences searched for their assigned paper online. Third, the reviewer may be able to guess the identity of the authors based on the content of the paper. The study~\cite{goues2018effectiveness} asked reviewers in three conferences which anonymized author identities to guess the authors of papers.
A total of 70\%-86\% of the reviews did not provide any guess (which could mean that the reviewer did not have any idea of the identity or that the reviewer chose to just not answer the question). Among the subset of reviews which contained a guess, 72\%-85\% guessed at least one author correctly. Some other works~\cite{shawndra03identification,caragea2019myth,matsubara2020citations} investigate the possibility of identifying authors based on content by designing machine learning algorithms to predict the authors based on the paper's content. In our experiment, we also investigate this aspect in ITCS 2023 by asking reviewers to self-report whether they can identify the authors.

Opening up author identities to reviewers near the end of the review process has been employed before in other computer science conferences~\cite{Hicks_2012}. There, the author identities are usually revealed right before the program committee meeting. Their stated reasons for this approach are that fully anonymous review might penalize a paper for failing to cite a certain prior work but this prior work may be unpublished work by the same authors, and anonymous review may not discover certain conflicts which the reviewer may know about. Our middle ground approach is also motivated by the other often stated use cases of author identities (Section~\ref{SecIntro}). Most importantly, a key objective of this approach in ITCS 2023 was to understand the use of author identities by reviewers to enable subsequent policy-makers to design policies using this evidence.

Questions about bias might arise more when decisions can be subjective, for instance, relying on criteria such as predictions of potential impact or novelty or interestingness to rank papers. There are various experiments in computer science and elsewhere that have found high disagreement among reviewers in terms of which paper is better~\cite{obrecht2007examining, fogelholm2012panel,nips14experiment,pier2017your,beygelzimer2021neuripsconsistency}, and more recently, experiments that have also found high disagreements among co-authors about their jointly authored papers~\cite{rastogi2022authors}. Consequently, there have been various debates about the low acceptance rates and ``selectivity'' in computer science conferences~\cite{fortnow2009viewpoint,parhami2016low,vardi2017divination}.

Our work focuses on biases associated with author identities. In addition to this, there are a variety of other issues in peer review. See~\cite{shah2022surveyextended} for details. These issues lead to a variety of challenging theoretical problems which can lead to considerable practical impact if solved, but are not well explored in theoretical computer science.

\section{Main results 1: Analysis of reviews}
In this section, we analyze the reviews to understand their dependence on author identities. We begin with a brief overview and statistics of the review process at ITCS 2023, and then present our analysis of the change in the reviews after the reviewer could see the author identities.\footnote{See \url{https://aspredicted.org/ng77i.pdf} for the preregistration of this analysis.}

\subsection{Overview and basic statistics of the review process}
\label{SecBasicReview}

The ITCS 2023 conference received a total of 235 paper submissions.
The set of reviewers comprised a program committee (PC) and external reviewers. The 40-member program committee were ultimately responsible for the acceptance/rejection decisions on all papers.  The program committee also included the program chair of the conference. The members of the program committee could review papers themselves or ask experts not in the program committee, called external reviewers, to review. In ITCS 2023, there were 40 members in the program committee who did a total of 405 reviews. There were 315 external reviewers who did 361 reviews.

The conference asked authors to anonymize their identities on the submitted paper. The conference policy did not prohibit authors from posting their (non anonymous) papers online or on social media or give talks. Reviewers were not shown the authors' identities until they submitted a review. After a reviewer submitted their review, they could see the authors' identities for that paper, and had the option to modify their review in any manner they chose. The reviewers were told of this policy beforehand, and were also told that the (changes in the) reviews will be analyzed.

Reviewers were asked to fill out following form when submitting their review for any paper:

\noindent\rule{\textwidth}{1pt}
\begin{itemize}
\item Overall merit: 5. Strong accept, 4. Accept, 3. Weak accept, 2. Weak reject, 1. Reject
\item Reviewer expertise: 4. Expert, 3. Knowledgeable, 2. Some familiarity, 1. No familiarity
\item Paper summary $\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_$
\item Comments for authors $\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_$
\item Comments for PC $\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_$
\item Author identity: Please indicate your knowledge of author identities. [Visible to program chairs. Not visible to authors at any time.]
    \begin{itemize}
        \item[A.] I know the author identities (e.g., saw a talk or on arxiv etc.)
        \item[B.] I have a reasonable guess for author identities
        \item[C.] I have no idea about author identities
    \end{itemize}
\end{itemize}
\noindent\rule{\textwidth}{1pt}

In Figure~\ref{FigReviewStatistics}, we plot the basic statistics pertaining to the reviews -- distribution of the overall merit scores, the self-reported reviewer expertise, and a histogram of the review length. Note that the reviewers could edit their reviews after they submit an initial version, and these statistics pertain to the final state of each review. The mean length of the reviews was 480 words and median was 425 words.

\begin{figure*}
\centering
\hspace{-1.2cm}
\begin{subfigure}{.24\textwidth}
\begin{tikzpicture}
\begin{axis} [xtick={0, 1,2,3,4},
        width=100pt,
        height=2in,
        ylabel = {\% Reviews},
        ymin=0, ymax=50,
         nodes near coords,
        xmin=-0.5,xmax=4.5,
        xticklabels={1,2,3,4,5},
        xticklabel style   = {align=center},
        every axis plot/.append style={
          ybar,
          bar width=10pt,
          bar shift=0pt,
          fill
        },
        ymajorgrids=true,
    ]
\addplot[fill = blue] coordinates{
    (0,9)
};
\addplot[fill = blue] coordinates{
    (1,28)
};
\addplot[fill = blue] coordinates {
    (2,31)
};
\addplot [fill=blue] coordinates {
    (3,29)
};
\addplot [fill=blue] coordinates {
    (4,3)
};
\end{axis}
\end{tikzpicture}
\caption{Overall merit}
\end{subfigure}
\hspace{-.3cm}
\begin{subfigure}{.24\textwidth}
\begin{tikzpicture}
\begin{axis} [xtick={0, 1,2,3},
        width=90pt,
        height=2in,
        ylabel = {\% Reviews},
        ymin=0, ymax=50,
         nodes near coords,
        xmin=-0.5,xmax=3.5,
        xticklabels={1,2,3,4},
        xticklabel style   = {align=center},
        every axis plot/.append style={
          ybar,
          bar width=10pt,
          bar shift=0pt,
          fill
        },
        ymajorgrids=true,
    ]
\addplot[fill = yellow] coordinates{
    (0,6)
};
\addplot[fill = yellow] coordinates{
    (1,31)
};
\addplot [fill=yellow] coordinates {
    (2,44)
};
\addplot [fill=yellow] coordinates {
    (3,19)
};
\end{axis}
\end{tikzpicture}
\caption{Reviewer expertise}
\end{subfigure}
\hspace{-.5cm}
\begin{subfigure}{.45\textwidth}
\begin{tikzpicture}
\begin{axis} [xtick={0, 17, 33, 50, 66, 83, 100},
        width=3.5in,
        height=2in,
        ylabel = {\# Reviews},
        ymin=0, ymax=50,

        xmin=-0.5,xmax=100.5,
        xticklabels={0,500,1000,1500,2000,2500,3000},
        xticklabel style   = {align=center},
        every axis plot/.append style={
          ybar,
          bar width=1pt,
          bar shift=0pt,
          fill
        },
        ymajorgrids=true,
    ]
\addplot[fill = black] coordinates
(0,7)(1,10)(2,18)(3,25)(4,19)(5,18)(6,37)(7,28)(8,21)(9,24)(10,43)(11,31)(12,45)(13,38)(14,39)(15,39)(16,28)(17,27)(18,30)(19,21)(20,15)(21,22)(22,16)(23,16)(24,12)(25,14)(26,10)(27,10)(28,5)(29,17)(30,8)(31,7)(32,2)(33,5)(34,1)(35,7)(36,1)(37,1)(38,3)(39,2)(40,2)(41,3)(42,2)(43,1)(44,2)(45,1)(47,2)(49,2)(50,3)(52,1)(53,1)(54,2)(98,1)(99,1)
};
\end{axis}
\end{tikzpicture}
\caption{Review length (number of words)}
\end{subfigure}
\caption{Basic statistics of the reviews in ITCS 2023.\label{FigReviewStatistics}}
\end{figure*}

We now come to the question about knowledge of author identities. It is important to note that this question was asked at the time the reviewer was submitting the review. This allows their answer to be based on a careful reading of the paper, and not just the title or abstract or a brief glance at the paper. In Figure~\ref{FigAutIde} we plot the distribution of reviewers' answers to this question. We observe among both program committee members and external reviewers, {\bf a majority of respondents report having no idea of the identities of the authors}.

\begin{figure*}
\centering
\begin{subfigure}{.4\textwidth}
\begin{tikzpicture}
\begin{axis} [xtick={0, 1,2},
        width=3in,
        height=2in,
        ylabel = {\% Reviews},
        ymin=0, ymax=100,
         nodes near coords,
        xmin=-0.5,xmax=2.5,
        xticklabels={Know,Guess,No idea},
        xticklabel style   = {align=center},
        every axis plot/.append style={
          ybar,
          bar width=30pt,
          bar shift=0pt,
          fill
        },
        ymajorgrids=true,
    ]
\addplot[fill = red] coordinates{
    (0,20)
};
\addplot[fill = orange] coordinates{
    (1,13)
};
\addplot[fill = green] coordinates {
    (2,67)
};
\end{axis}
\end{tikzpicture}
\caption{Program committee members}
\end{subfigure}
\qquad\qquad
\begin{subfigure}{.4\textwidth}
\begin{tikzpicture}
\begin{axis} [xtick={0, 1,2},
        width=3in,
        height=2in,
        ylabel = {\% Reviews},
        ymin=0, ymax=100,
         nodes near coords,
        xmin=-0.5,xmax=2.5,
        xticklabels={Know,Guess,No idea},
        xticklabel style   = {align=center},
        every axis plot/.append style={
          ybar,
          bar width=30pt,
          bar shift=0pt,
          fill
        },
        ymajorgrids=true,
    ]
\addplot[fill = red] coordinates{
    (0,25)
};
\addplot[fill = orange] coordinates{
    (1,21)
};
\addplot[fill = green] coordinates {
    (2,54)
};
\end{axis}
\end{tikzpicture}
\caption{External reviewers}
\end{subfigure}
\caption{Distribution of reviewers' self reports about their knowledge of author identities.\label{FigAutIde}}
\end{figure*}

\subsection{Change in reviews after initial submission}
\label{SecChangeAnalysis}

Recall that the reviewers could not see the identities of the authors of the paper initially, but the identities were made visible to them after they submitted a review for the paper. The reviewers could then edit their reviews, and in this section, we analyze the changes to the reviews after initial submission.

Before delving into the details, we must address one confounder. Once a program committee member submitted their review, they could also see all other reviews submitted for that paper. Thus any change in their review may be due to other reviews, and such an influence of other reviews is well documented in the literature~\cite{teplitskiyasocial}. Consequently, for program committee members, we restrict attention to only those reviews that were the first to be submitted for the paper. In other words, in this analysis, we only consider a review by a program committee member if there was no other review submitted for that paper when this review was first submitted.  The external reviewers could not see other reviews at any time, and hence we consider all external reviews. This leaves us with 90 reviews by program committee members, along with the 361 reviews by external reviewers for our analysis. In the remainder of this section, we restrict attention to these 451 reviews.

We tabulate the percentage and number of reviews that changed in Table~\ref{TabNumChanged}. Observe that the number of reviews that changed the overall merit and reviewer expertise is {\bf very small}. Also observe that among the reviews that changed, a sizable fraction indicated that they knew the author identities, which suggests that either these reviews are changed for reasons not having to do with author identities, or that initially hiding the identities from the paper had some effect despite the reviewers' knowledge of author identities from elsewhere. Among the 32 reviews that changed overall merit, the mean change was $-0.4$ and median change was $-1$, and among the 17 reviews that changed reviewer expertise, the mean change was $+0.47$ and median was $+1$.

We now delve deeper into the changes in overall merit and reviewer expertise, where we compute the correlation between these changes and the rank of the affiliations of the authors of the paper. We compute the rank of any affiliation based on \url{csrankings.org} (2012 to 2022, Theory $\rightarrow$ Algorithms and Complexity). Affiliations that were not listed on csrankings (such as industry labs) where ranked by the program chair. Then for any paper, we take the best rank among all affiliations associated with the authors of that paper.

Restricting attention to the 32 reviews where the overall merit was changed after initial submission, we compute the Kendall tau b correlation between the change in overall merit and the rank of the associated paper. A positive correlation means that the overall merit for a paper with a better rank increased after the initial submission. We find that even after restricting to only the reviews that changed, {\bf the correlation is very weak, with a correlation coefficient of 0.09}.

Next we look at the 17 reviews where the reviewer expertise was changed after initial submission. In these 17 reviews, we wish to check if the self-reported reviewer expertise increased if the paper was from a better-ranked affiliation and the reviewer gave a high overall merit score, or if the paper was from a lower-ranked affiliation and the reviewer gave a low overall merit score. In what follows, this would be implied by a positive correlation coefficient and the opposite by a negative correlation coefficient. We compute the Kendall tau b correlation between the affiliation-rank of the paper and the quantity (updated expertise - old expertise)*(1 if overall merit=accept and -1 if overall merit=reject). We find that the coefficient is 0.46, meaning that restricting attention to these reviews we find a moderate effect, although we emphasize that this pertains only to a small number of reviews which actually changed.

Finally, in 5 of the 315 reviews by external reviewers, the reviewer changed their answer about knowledge of author identities after initial submission. In two of these cases, they changed from `no idea' to `know', one changed from `know' to `no idea', one from `guess' to `no idea', and one from `no idea' to `guess'. None of the 90 reviews by program committee members changed the answer to the question on knowledge of author identities.

\begin{table}
\centering
\begin{tabular}{|l|c|c|c|}
    \hline
     & Author identity & Change  in overall merit & Change in reviewer expertise \\\hline
     Program & Know &1/20 = 5.0\% & 1/20 = 5.0\%  \\
     committee & Guess&1/11 = 9.1\% & 0/11 = 0\%  \\
     member& No idea &4/59 = 6.8\% & 1/59 = 1.7\%  \\\hline
     External & Know &5/90 = 5.6\% & 6/90 = 6.7\%  \\
     reviewer & Guess&6/76 = 7.9\% &  4/76 = 5.2\%  \\
     & No Idea &15/195 = 7.7\% & 5/195 = 2.6\%  \\\hline
     Total &  & 32/451 = 7.1\% & 17/451 = 3.8\%  \\\hline
\end{tabular}
\caption{Number and percentage of reviews that changed after initial submission.\label{TabNumChanged}}
\end{table}

\section{Main results 2: Survey}
\label{SecSurvey}

We conducted an anonymous, optional survey among the participants of the ITCS 2023 conference. The survey was sent out via email on November 2, 2022 (soon after the paper acceptance decisions were announced) and was open till November 15, 2022. We first present the contents of the survey  as shown to the participants, and then present aggregated responses.

\subsection{Questionnaire presented to participants}
In what follows, we present the survey questionnaire verbatim as presented to the participants.

\noindent\rule{\textwidth}{1pt}
Our field has long debated whether to anonymize authors in the review process. The main reported benefit of anonymizing authors is that of reducing (biased) dependence on author identities in the review, as has been found in a number of experiments in other fields. A number of cons are also often discussed:
\begin{itemize}
\item Challenges in conflict-of-interest detection
\item Reviewers want to use author identities for some part of their review (e.g., using track record to gain confidence in whacky ideas or complex proofs)
\item Reduces ambiguity w.r.t. prior literature (e.g., can know that a preprint was by the same authors)
\item Imperfectness of anonymization.
\end{itemize}

ITCS 2023 adopted a middle ground where reviewers initially did not see author identities, but could see after submitting their initial reviews, and could then modify their reviews. We would love to know your thoughts on this debate of anonymizing authors. When answering, please assume that conflict-of-interest detection is taken care of even if authors are anonymized (as done in many other conferences that adopt author anonymization).

~\\

\noindent I participated in ITCS 2023 as:
\begin{itemize}
 \renewcommand{\labelitemi}{$\square$}
    \item PC member
    \item External reviewer
    \item Author
\end{itemize}

\noindent Preferences regarding anonymizing authors:\footnote{The ordering of the first four options was randomized to be either this order or reverse order.}
\begin{itemize}
 \renewcommand{\labelitemi}{$\square$}
    \item I prefer anonymizing author identities through the entire review process
    \item I prefer anonymizing author identities until the time for PC meetings/discussions
    \item I prefer anonymizing author identities until the reviewer submits their initial review
    \item I prefer not anonymizing author identities
    \item Other (please specify) $\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_$
\end{itemize}

\noindent
Please share the reasons for your preferences, as well as any other comments/experiences/opinions on this topic: $\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_$

\noindent\rule{\textwidth}{1pt}

\subsection{Analysis of survey responses}
We received 200 responses to the questionnaire.  In this section, we summarize and analyze these responses.

\definecolor{colorA}{HTML}{003f5c}
\definecolor{colorD}{HTML}{8a61a5}
\definecolor{colorI}{HTML}{ef5675}
\definecolor{colorN}{HTML}{ffa600}
\definecolor{colorO}{HTML}{ffffff}

\newcommand{}{}
\newcommand{north east lines}{north east lines}
\newcommand{fivepointed stars}{fivepointed stars}
\newcommand{bricks}{bricks}
\newcommand{dots}{dots}

\begin{figure*}
\centering
\begin{subfigure}{\textwidth}
\begin{tikzpicture}
\begin{axis} [xtick={0, 1,2,3,4},
        width=3in,
        height=2in,
        ylabel = {\# Responses},
        ymin=0, ymax=90,
         nodes near coords,
        xmin=-.5,xmax=4.5,
        xticklabels={,,,,},
        xticklabel style   = {align=center},
        every axis plot/.append style={
          ybar,
          bar width=30pt,
          bar shift=0pt,
          fill
        },
        ymajorgrids=true,
        legend style={at={(2.5,0.5)}, anchor=east,legend columns=1},
        legend image code/.code={
        \draw [#1] (1cm,-.3cm) rectangle (0.25cm,0.25cm); },
        legend cell align={left}
    ]
\addplot[fill = colorA, postaction={pattern=}] coordinates{
    (0,77)
};
\addplot[fill = colorD, postaction={pattern=north east lines}] coordinates{
    (1,29)
};
\addplot[fill = colorI, postaction={pattern=fivepointed stars}] coordinates {
    (2,58)
};
\addplot [fill=colorN, postaction={pattern=bricks}] coordinates {
    (3,52)
};
\addplot [fill=colorO, postaction={pattern=dots}] coordinates {
    (4,12)
};
\legend{Anonymizing through entire review process,Anonymizing until PC meetings/discussions,Anonymizing until reviewer submits initial review, Not anonymizing author identities, Other}
\end{axis}
\end{tikzpicture}
\caption{All responses (200 responses)\label{FigSurveyAll}}
\end{subfigure}\\~\\
\begin{subfigure}{.35\textwidth}
\begin{tikzpicture}
\begin{axis} [xtick={0, 1,2,3,4},
        width=2.3in,
        height=2in,
        ylabel = {\# Responses},
        ymin=0, ymax=90,
         nodes near coords,
        xmin=-.5,xmax=4.5,
        xticklabels={,,,,},
        xticklabel style   = {align=center},
        every axis plot/.append style={
          ybar,
          bar width=20pt,
          bar shift=0pt,
          fill
        },
        ymajorgrids=true
    ]
\addplot[fill = colorA, postaction={pattern=}] coordinates{
    (0,6)
};
\addplot[fill = colorD, postaction={pattern=north east lines}] coordinates{
    (1,1)
};
\addplot[fill = colorI, postaction={pattern=fivepointed stars}] coordinates {
    (2,4)
};
\addplot [fill=colorN, postaction={pattern=bricks}] coordinates {
    (3,9)
};
\addplot [fill=colorO, postaction={pattern=dots}] coordinates {
    (4,3)
};
\end{axis}
\end{tikzpicture}
\caption{Program committee (PC) members (19 responses)\label{FigSurveyPC}}
\end{subfigure}
~
\begin{subfigure}{.3\textwidth}
\begin{tikzpicture}
\begin{axis} [xtick={0, 1,2,3,4},
        width=2.3in,
        height=2in,
        ymin=0, ymax=90,
         nodes near coords,
        xmin=-.5,xmax=4.5,
        xticklabels={,,,,},
        xticklabel style   = {align=center},
        every axis plot/.append style={
          ybar,
          bar width=20pt,
          bar shift=0pt,
          fill
        },
        ymajorgrids=true
    ]
\addplot[fill = colorA, postaction={pattern=}] coordinates{
    (0,22)
};
\addplot[fill = colorD, postaction={pattern=north east lines}] coordinates{
    (1,16)
};
\addplot[fill = colorI, postaction={pattern=fivepointed stars}] coordinates {
    (2,36)
};
\addplot [fill=colorN, postaction={pattern=bricks}] coordinates {
    (3,17)
};
\addplot [fill=colorO, postaction={pattern=dots}] coordinates {
    (4,4)
};
\end{axis}
\end{tikzpicture}
\caption{External reviewers\\(84 responses)\label{FigSurveyExternal}}
\end{subfigure}
~
\begin{subfigure}{.3\textwidth}
\begin{tikzpicture}
\begin{axis} [xtick={0, 1,2,3,4},
        width=2.3in,
        height=2in,
        ymin=0, ymax=90,
         nodes near coords,
        xmin=-.5,xmax=4.5,
        xticklabels={,,,,},
        xticklabel style   = {align=center},
        every axis plot/.append style={
          ybar,
          bar width=20pt,
          bar shift=0pt,
          fill
        },
        ymajorgrids=true
    ]
\addplot[fill = colorA, postaction={pattern=}] coordinates{
    (0,49)
};
\addplot[fill = colorD, postaction={pattern=north east lines}] coordinates{
    (1,12)
};
\addplot[fill = colorI, postaction={pattern=fivepointed stars}] coordinates {
    (2,18)
};
\addplot [fill=colorN, postaction={pattern=bricks}] coordinates {
    (3,26)
};
\addplot [fill=colorO, postaction={pattern=dots}] coordinates {
    (4,5)
};
\end{axis}
\end{tikzpicture}
\caption{Authors excluding external reviewers and PC (97 responses)\label{FigSurveyAuthors}}
\end{subfigure}
\caption{Answers given by respondents to the prompt ``Preferences regarding anonymizing authors.''\label{FigSurvey}}
\end{figure*}

\subsubsection{Quantitative analysis}
\label{SecQuantitative}
We begin with an analysis of the responses to the quantitative question on the respondents' preferences about anonymizing authors. We report the results in Figure~\ref{FigSurvey}.\footnote{The participants were allowed to choose more than one option. Here we report the total counts in terms of the number of respondents who chose any option (hence the sum of all options in Figure~\ref{FigSurvey} is greater than 200). An alternative approach would be as follows: If a participant chooses $k$ options, then count each chosen option as $\frac{1}{k}$. The result from this alternative approach is qualitatively similar to the current results, and hence we omit it for brevity.}  The key takeaway is that there is considerable support for anonymizing authors in at least some part of the review process, particularly from participants outside the program committee.

Notice in Figure~\ref{FigSurveyAll} that twelve respondents chose the option `other.' These respondents also provided comments alongside this choice. The three PC members who selected `other' also indicated support for anonymization in some form, such as anonymization to reviewers but not to program committee members. Five of the twelve respondents who selected `other' and did not select any other option. Three of these five were supportive of anonymizing authors in peer review, but were concerned about challenges in detecting conflicts of interest (CoI). In absence of other ways for CoI detection, they supported anonymizing except for CoI detection. One of them strongly supported anonymization at least for external reviewers. One other respondent did not have an opinion. The remaining seven respondents who selected `other' also selected other options alongside, and are already counted in Figure~\ref{FigSurveyAll}. We discuss their text comments below along with all other general comments made by respondents.

\subsubsection{Free-text comments}

We finally discuss the free-text comments provided by respondents. A total of 103 respondents provided free-text comments, and in what follows we summarize these comments. We saw in Figure~\ref{FigSurvey} that there were disagreements among respondents on the best policies for anonymizing author identities, and these disagreements are also reflected in the free-text comments. In our summarization below, we will focus on the content of the comments rather than the pure opinion about which policies to use, which were already captured in Section~\ref{SecQuantitative}.

We begin with {\bf comments that were common to a large number of respondents}.
\begin{itemize}
    \item Respondents opine that revealing author identities can bias the reviews. For example, some representative comments are:
        \begin{itemize}
        \item ``There's ample evidence now that our judgements can be biased by aspects of individuals' identities, despite our best intentions otherwise. Anonymizing author identities should help with that.''
        \item ``This should be done to avoid bias. This also welcomes new authors to the field.''
        \end{itemize}
    \item Respondents appreciated the initiative taken by ITCS 2023 in adopting a middle-ground approach.  Some representative comments are:
        \begin{itemize}
        \item ``I think that hiding author identities helps well-intentioned reviewers reduce their bias. Having the author identities available later in the process allows one to deal with other issues (concurrent results, overlap in authorship with previous papers, etc.) more easily. ''
        \item ``I was somewhat surprised when I saw the author list, which I think is a good indication that it probably was good to withhold the information until after I submitted the review.''
        \item ``I think the middle ground policy we used this time is MUCH better than other more/less anonymous policies. It worked really well for me!!''
        \end{itemize}
    \item Respondents complained about problems in avoiding conflicts-of-interest when assigning external reviewers. Some representative comments are:
    \begin{itemize}
    \item ``It's challenging to use subreviewers when author identities are anonymous."
    \item ``I generally support author anonymity, but I had a bit of a tricky situation with a CoI as a result this time."
    \end{itemize}
\end{itemize}

\noindent We now discuss the remaining comments, each of which was given {\bf by one or few respondents}.  We have broadly classified these comments in terms of their implications regarding author anonymization.
\begin{itemize}
\item {\bf Problems with anonymization / benefits of knowing author identities}:
    \begin{itemize}
        \item Many authors upload their papers on preprint servers like arXiv or elsewhere, thereby resulting in imperfect anonymization.

        \item Anonymization makes rejecting a paper of a rival easier: A reviewer can identify the paper and reject, no one can blame the reviewer since it is ``anonymous".

        \item Some situations don't yet have suitable policies for anonymization, e.g., submitting a code repository on Github.

        \item In a small community (like ITCS), reviewers can easily guess the identities of authors.

        \item Even if anonymized, some biasing information about authors may be leaked by the English.

        \item Introducing anonymization signals mistrust in the integrity of our community and inevitably pushes people into being less trustworthy.

        \item Authors may strongly criticize their own past work to bolster the current submission. If the submission is anonymized, reviewers will not know that it is the same set of authors.

        \item It can be challenging for authors to ensure that they do not inadvertently identify themselves.

        \item Once it is realized that anonymization does not realize its intended goals, it may then lead to undesirable policies such as preprint publication and talk embargoes.

        \item Anonymization would create weird incentives around posting papers on preprint servers, where authors might delay posting papers to stay anonymous or post earlier to make their identities public, depending on what they think would benefit them.

        \item The best hedge against bias and bad reviews is a community standard of collegiality,  openness,  fairness,  and self control.   Such an atmosphere can  only   flourish  in an  environment where people are trusted  and everything is above board.

        \item Some authors have a history of making errors or submitting bad papers, so knowing author identities can help assess correctness concerns.

        \item Author identities can be used to assess incremental contributions:
         followup works by the same set of authors should be subject to higher bars than followup works by new authors.

        \item Papers by student authors should have lower bars, or in some cases can be shepherded instead of outright rejection.

        \item It is very hard to assess quality in real time, and author reputation can serve as a good distinguisher.

        \item Authors' track record provides additional confidence to reviewers, for instance, if they are not entirely familiar with the paper or if there are very long proofs.

        \item Author identities tell the reviewer how to read the paper stylistically.

        \item Reviewers should know who the authors are eventually, so that they can understand whom they should credit if they happen to learn something during the review process that they later (even unintentionally) end up using.

        \item If there are multiple closely related papers on a subject, it is important to understand to what extent these are distinct groups of authors in each combination or just closely related papers by very similar authorship teams.
\end{itemize}

\item {\bf Benefits of anonymizing}:

\begin{itemize}
    \item Anonymization welcomes new authors to the field.

    \item ``I trusted my evaluation more when I did it without knowing who the authors were.''

    \item Anonymization helps bring new ideas to the conference.

    \item Ideas and proofs should talk for themselves, and authors should explain new ideas well. Author name should not be a substitute.

    \item It will force reviewers to read the paper carefully rather than judging the quality based on authors.

    \item It is for the papers in the grey area that author identity will bias decisions, and these papers are precisely the ones for which the discussion is critical.

    \item  Dependence on author identities to review makes our community unwelcoming and closed to others.

    \item Revealing author identities may make the reviewer more careful if the reviewer knows that the identities will be known at the discussion phase.

\end{itemize}

\item {\bf Benefits of middle ground}:
\begin{itemize}
    \item Prevents initial bias but then helps reap benefits of not anonymizing, thereby achieving both goals.

    \item People get curious and will look up the paper,  but they will be patient enough to wait until after the first review if they know they will get a chance to then see and potentially update their review.

    \item Does not do any harm (presuming the initial review is also saved and the PC can see the changes made) and is good as a backup option.

    \item Anonymity until PC meetings will ensure that the reviewers are not biased. Revealing identities during discussions can rule out confusion regarding originality of the work.

    \item People work hard to cultivate a good reputation for high quality impactful research. This middle-ground mitigates implicit discrimination and that explicit discrimination is hopefully not too wide-spread, and anyhow very hard to deal with.

    \item Allows those committee members who want to ignore authorship to do so when writing their reviews, while not constraining others from using it.
\end{itemize}

\item {\bf Problems with middle ground}:
\begin{itemize}
    \item Puts back the bias into the process.

    \item If a reviewer has initially given a negative review to famous authors, after finding the reviewer identities, a reviewer may be tempted to change the review.
\end{itemize}

\item {\bf Suggestions}:
\begin{itemize}
    \item The program committee members should be able to see the name in the conference management platform but not on the paper.

    \item Ask the reviewers to give two scores, one on the quality of the paper/result/novelty and another on confidence in the correctness/technical part. Among them, only the ``correctness confidence score" can be modified after seeing the authors' names after submitting an initial review.

    \item It should be the reviewer's choice as to whether they want author identities anonymized or not.

    \item The elimination of author names from the front page is a simple change to the paper.

    \item Prohibiting arXiv and other external dissemination is not good, but there is no harm in anonymizing within the review process.

    \item Can have a two-tiered PC where the senior PC are aware of the author identities but do not actively participate in reviewing, but helps to resolve any issues with conflicts of interest.

    \item In this year's data with the middle-ground approach, it may be possible to test the bias in the process after initial review submission. Find the correlation between author's fame or affiliation ranking and the acceptance decisions, conditioned on the initial review scores.\footnote{This is certainly an interesting suggestion. However, it may lead to violation of false alarm requirements. To see this, consider a simplified scenario where papers are either from influential or non-influential authors, and where papers by influential authors actually have a higher ``true'' quality. Suppose the true quality of papers by influential authors is 4 out of 5, and that of papers by non-influential reviewers is 2 out of 5. The review process is noisy, and suppose the mean score given to any paper is its true score+1 with probability 0.5 and true score-1 otherwise. Finally, in this model, the discussion process simply serves to denoise the scores, thereby accepting papers with a higher true score. There is thus no bias in the entire process. Now, the data will comprise a set of papers -- some from influential authors and others from non-influential authors -- each of which has a mean reviewer score of 3. However, the acceptance rates for papers with more influential authors will be higher, thereby leading such an analysis to falsely conclude existence of a bias. Designing statistical tests which can detect this effect with a guaranteed control on false alarm is an interesting question for future work.}

    \item In the middle ground, PCs should be able to see any change in the reviews once the reviewer sees the author identities.
\end{itemize}

\item {\bf Other comments}:
\begin{itemize}
    \item Anonymization of author identities is already adopted in most subfields of computer science.

    \item The real problem is that the heavy emphasis on conference publications which can only be refereed incompletely, and far too few papers end up going through a rigorous refereeing process by a journal.

    \item We should as a community be more careful about who we pick as reviewers and make sure there is some diversity in that process as well. It would be good to gather statistics on the demographics of reviewers.  Perhaps a reminder that statistics will be gathered at the end of the process would be enough to improve things.
\end{itemize}
\end{itemize}

\section{Discussion and limitations}
Our results suggest support for some form of anonymizing authors in peer review. Doing so can help mitigate biases pertaining to initial impressions about authors. In the literature, there are some stated benefits of not anonymizing authors, such as using author identities to gain confidence in a proof or wacky idea, and also to be circumspect of authors who have a history of submitting low-quality work. While these stated benefits are debated in the survey responses, if a venue wishes to realize them, the approach followed by ITCS 2023 may allow them to do so. Although this may result in bias appearing in the process after revelation of author identities, in ITCS 2023, we did not find much change in the overall merit scores after author identities were revealed. (Note that we were unable to check changes in review text or eventual discussions, leaving open the possibility of author identities playing a role there.) It is also sometimes stated that anonymizing authors is not useful as author identities may be guessed from the contents of the paper. However, we found that a majority of reviewers were unable to guess the authors' identities.

Another stated benefit of knowing author identities is the ease of checking conflicts of interest. This was indeed a challenge at ITCS 2023, where several participants in the survey complained about problems in ensuring that papers are assigned to external reviewers who do not have conflicts of interest with the paper. Thus an anonymization of authors should be accompanied by an efficient and rigorous process to check conflicts of interest. One way to do so is to have an automated system to check for co-authorship and affiliation conflicts, as is done in machine learning conferences. An alternative ``manual'' option is to have a small set of volunteers who can check conflicts for any external reviewer that a program committee member wishes to invite for reviewing. A third possible solution, which leads to some reduction in anonymity, is to make author identities anonymous only to external reviewers but not program committee members.

Note that importantly, the conference did not impose any restrictions on authors regarding posting their (non anonymous) papers elsewhere such as on preprint servers. Such policy choices are widely debated~\cite{rastogi2022arxiv} in some other research communities, where arguments in favor of such restrictions point to ensuring more thorough anonymization, whereas arguments against such restrictions include allowing free and open dissemination of research and furthermore of unintentionally biasing against the very people that it is supposed to protect~\cite[Chapter 7]{shah2022surveyextended}. The policies of ITCS 2023, as well as the analysis and discussion in the present paper pertain to the absence of such restrictions.

This study also has several limitations that we discuss below.
\begin{itemize}
    \item As briefly mentioned above, we did not have access to any changes in the text of the reviews.\footnote{Our preregistration did include analysis of the change of text. However, we could not access the logs of the review text from the HotCRP conference management platform on which the peer-review process was conducted.} Author identities may also have played a role in the discussions between program committee members but we do not have logs of these discussions to draw any inference.
    \item One manner in which reviewers may use author identities is to gain confidence in their evaluation. We attempt to measure this, however, the review questionnaire asked reviewers to self report their expertise. The confidence and expertise of any reviewer may be related, but not identical. For instance, a self-report of expertise may pertain to the topic of the paper whereas a self-report of confidence may pertain to their evaluation of the paper.
    \item The survey we conducted was anonymous, and hence comes with the usual caveats associated with anonymous surveys such as possible selection biases. We do stratify the responses by the respondents self-reported role in ITCS 2023 (program committee member, external reviewer, or author) but investigation of selection biases or stratification along other attributes is not possible.

    \item The experiment was announced to reviewers prior to them starting to review, and they were told that the logs of the review revisions will be analyzed. It is thus possible that this information may have changed the reviewer's behavior (Hawthorne effect).
\end{itemize}
As for the last point above, we think that future conferences adopting such policies of partial anonymization should consider imparting transparency with respect to dependence on author identities, in which the program chairs, or the program committee, or even the authors can see the revisions of any review.

All in all, we hope that this investigation will lead to more discussion and evidence-based policy design for an improved peer-review process.

\section*{Acknowledgments}
We sincerely thank the ITCS 2023 program chair Yael Kalai for trying out this middle ground approach to author anonymization, for facilitating the analysis, and for  valuable inputs throughout the analysis. We also thank Pravesh Kothari for very helpful discussions. We are grateful to all the program committee members and reviewers of ITCS 2023 for their efforts in the review process, and all the respondents of our survey for sharing their opinions and suggestions. This work was reviewed and approved by the CMU Institutional Review Board (IRB).

\let\OLDthebibliography\thebibliography
\renewcommand\thebibliography[1]{
  \OLDthebibliography{#1}
  \setlength{\parskip}{5pt}
  \setlength{\itemsep}{0pt plus 0.3ex}
}
\printbibliography

\end{document}

redpajama_set_name: RedPajamaArXiv


With the software RP Fiber Power one can simulate how channel powers evolve in a system, how cross-talk arises from nonlinear interactions, etc.
Wavelength division multiplexing is a kind of frequency division multiplexing – a technique where optical signals with different wavelengths are combined, transmitted together, and separated again. It is mostly used for optical fiber communications to transmit data in several (or even many) channels with slightly different wavelengths. In this way, the transmission capacities of fiber-optic links can be increased strongly, so that most efficient use is made not only of the fibers themselves but also of the active components such as fiber amplifiers. Apart from telecom, wavelength division multiplexing is also used for, e.g., interrogating multiple fiber-optic sensors within a single fiber.
Theoretically, the full data transmission capacity of a fiber could be exploited with a single data channel of very high data rate, corresponding to a very large channel bandwidth. However, given the enormous available bandwidth (tens of terahertz) of the low-loss transmission window of silica single-mode fibers, this would lead to a data rate which is far higher than what can be handled by optoelectronic senders and receivers. Also, various types of dispersion in the transmission fiber would have very detrimental effects on such wide-bandwidth channels, so that the transmission distance would be strongly restricted. Wavelength division multiplexing solves these problems by keeping the transmission rates of each channel at reasonably low levels (e.g. 10 Gbit/s or 100 Gbit/s) and achieving a high total data rate by combining several or many channels.
Coarse wavelength division multiplexing (CWDM, ITU standard G.694.2 ) uses a relatively small number of channels, e.g. four or eight, and a large channel spacing of 20 nm. The nominal wavelengths range from 1310 nm to 1610 nm. The wavelength tolerance for the transmitters is fairly large, e.g. ±3 nm, so that unstabilized DFB lasers can be used. The single-channel bit rate is usually between 1 and 3.125 Gbit/s. The resulting total data rates are useful e.g. within metropolitan areas, as long as broadband technologies are not widespread in households (→ fiber to the home).
Dense wavelength division multiplexing (DWDM, ITU standard G.694.1 ) is the extended method for very large data capacities, as required e.g. in the Internet backbone. It uses a large number of channels (e.g. 40, 80, or 160), and a correspondingly small channel spacing of 12.5, 25, 50 or 100 GHz. All optical channel frequencies refer to a reference frequency which has been fixed at 193.10 THz (1552.5 nm). The transmitters have to meet tight wavelength tolerances. Typically, they are temperature-stabilized DFB lasers. The single-channel bit rate can be between 1 and 100 Gbit/s, and in the future even higher.
Due to the wide amplification bandwidth of erbium-doped fiber amplifiers, all channels can often be amplified in a single device (except in cases where e.g. the full range of CWDM wavelengths is used). However, problems can arise from the variation of gain with wavelength or from interaction of the data channels (crosstalk, channel interference) e.g. via fiber nonlinearities. Enormous progress has been achieved with a combination of various techniques, such as the development of very broadband (double-band) fiber amplifiers, gain flattening filters, nonlinear data regeneration and the like. The system parameters such as channel bandwidth, channel spacing, transmitted power levels, fiber and amplifier types, modulation formats, dispersion compensation schemes, etc., need to be well balanced to achieve optimum overall performance.
Even for existing fiber links with only one or a few channels per fiber, it can make sense to replace senders and receivers for operation with more channels, as this can be cheaper than replacing the whole system with a system with a higher transmission capacity. In fact, this approach often eliminates the need to install additional fibers, even though the demand on transmission capacities is increasing enormously.
Apart from increasing the transmission capacity, wavelength division multiplexing also adds flexibility to complex communication systems. In particular, different data channels can be injected at different locations in a system, and other channels can be extracted. For such operations, add–drop multiplexers can be used, which allow one to add or drop data channels based on their wavelengths. Reconfigurable add–drop multiplexers make it possible to reconfigure the system flexibly so as to provide data connections between a large number of different stations.
In many cases, time division multiplexing (TDM) can be an alternative to wavelength division multiplexing. Here, different channels are distinguished by arrival time rather than by wavelength.
Technica provides several of the key components used in wavelength division multiplexing solutions including fiber Bragg gratings (Txxx family of products), Fabry–Pérot etalons and tunable filters (TFPxxx family of products), and athermal references including HCN and C2H2 gas cells (TWRxxx family of products).


text:
The Top 10 Premier League Goalscorers In The 2010's
Last updated 3:59 PM, Friday December 20 2019 GMT
With the 2010's coming to an end, Sergio Aguero will end the decade as the highest-scoring top scorer in the Premier League.
The Argentine, who joined Manchester City in the summer of 2011 from Atletico Madrid, has scored a ridiculous 173 goals in the top flight.
He scored 23 in his debut season - his final strike being that infamous last-gasp goal against QPR to win City's first league for 44 years - and has remained incredibly prolific throughout his time in England.
With persistent injuries, 'Kun' has only failed to hit 20 goals in two separate seasons - in 2012/13 and 2013/14, and won the Golden Boot in 2014/15 when he bagged 26 in the league.
Now 31, Aguero is comfortably the top Premier League scorer of the decade but Harry Kane's excellent goalscoring record sees him in second on 134 goals - which is incredibly impressive considering he wasn't a regular in the Spurs side until 2014.
His lowest league tally was 17 - and given he's still only 26 - he's easily the most likely candidate to level Alan Shearer's overall record.
Wayne Rooney's goals dried up towards the end but his form at the start of the decade was enough to put him in third place on 114, one ahead of Romelu Lukaku, who bagged plenty for West Brom, Everton and Manchester United before his switch to Serie A.
Robin van Persie is fifth with 98 goals for Arsenal and Manchester United, with Jamie Vardy doing superbly to occupy sixth spot after starting the decade in the lower echelons of Non-League football.
The Leicester City hitman is just four goals from a Premier League century, having notched 16 in 17 this term for the Foxes.
Revealed: 12 Players That Received A 10/10 Rating In A Premier League Match This Season
The Premier League's Top Five Biggest Divers Have Been Named
Manchester United Are In A 'False Position' At The Top Of The Premier League
Mauricio Pochettino Names His Three Top Players In The Premier League
Manchester United Climb To The Top Of The Premier League Table After 1-0 Win Against Burnley
Jamie Vardy Wanted To Quit Leicester City And Become A Party Rep In Ibiza
Eden Hazard lies in seventh after scoring 85 goals, seven more than his former teammate Olivier Giroud in eight.
Raheem Sterling and Sadio Mane, two of the league's top marksmen these days, are level on 75 goals and complete the list thanks to their heroics in front of goal.
Here is the list in full:
The Players With Most Premier League Match-Winning Goals Have Been Named
1. Sergio Aguero (173 goals)
2. Harry Kane (134 goals)
3. Wayne Rooney (114 goals)
4. Romelu Lukaku (113 goals)
5. Robin van Persie (98 goals)
6. Jamie Vardy (96 goals)
7. Eden Hazard (85 goals)
8. Olivier Giroud (78 goals)
9. = Raheem Sterling (75 goals)
9, = Sadio Mane (75 goals)
Topics: jamie vardy, Football News, Sergio Aguero, Football, Premier League, Harry Kane

redpajama_set_name: RedPajamaCommonCrawl


text:
Huck Institutes seeks new director for Center of Excellence in Industrial Biotechnology
The Huck Institutes of the Life Sciences is seeking nominations, including self-nominations, for director of the Center of Excellence in Industrial Biotechnology (CoEIB). Consideration will begin on January 18 and will continue until the position is filled.
The Center of Excellence in Industrial Biotechnology (CoEIB) is supported by significant gift funds that underpin its three primary missions: workforce development, research, and outreach. This includes:
• Undergraduate students: scholarships, internships, talent acquisition programs, and the Society for Industrial Biotechnology
• Graduate students: scholarships, co-ops, biomanufacturing laboratory course
• Support for faculty and student research
• IMPACT seed grant program
• Requests for projects in collaborative core facilities
• Fermentation and cell culture workshops: Virtual and in-person
• Commonwealth campus programs
• Internal and external facility tours
Additional funds are available to support the Center's administration including partial salary support for the director, a full-time associate director and a part-time communications strategist. The Huck Institutes' director, as budgetary executive, approves expenditures by the CoEIB through the Huck business office.
The CoEIB operates in conjunction with the CSL Behring Fermentation Facility and the Cell Culture Facility forming an ecosystem that supports the Center's three mission-driven areas in industrial biotechnology. As part of this collaboration, the CoEIB director works with the facility's leadership and other faculty to develop industry-relevant student training programs and research, leveraging the state-of-the-art equipment found in the CSL Behring Fermentation Facility and the Cell Culture Facility.
The CoEIB faculty advisory committee, appointed by the CoEIB director in consultation with the Huck Institutes director, will be responsible for evaluating the performance of the CoEIB, monitoring expenditure of funds, and providing approval oversight for the engagement of future stakeholders.
The CoEIB director should hold a tenured Penn State faculty appointment or have equivalent prior experience. The successful nominee will report to the Huck Institutes director, be responsive to the CoEIB faculty advisory committee and external advisory board, assist with the acquisition of additional funds and champion industrial biotechnology initiatives at Penn State.
Nominees should send their CV, a one-page vision statement for the CoEIB and a cover letter highlighting their pertinent experience to Professor Andrew Zydney, (alz3@psu.edu).
A copy of the gift proposal is available for examination upon request. Nominations will be reviewed by an advisory committee, with interviews arranged as appropriate. Final decisions will be made by the director of the Huck Institutes.
Consideration begins on January 18 and continues until the position is filled.


Car Interior Accent Lights. car interior accent lighting 2017. 2011 nissa cube overview specs pictures. 6pc car interior neon underglow accent light kit campatible with iphone ipad and ipod touch. car accent lighting. xkchrome bluetooth app controlled 6pc interior car truck led accent light kit ebay. 12 inch 24 led light tube led strip lights accent lighting led modules super bright leds. 4x 12v blue led car auto interior dash floor decoration floor accent light lamp ebay. accent lighting for cars home design. nissan sentra interior accent lighting 2008 2012 999f3 lu000 aw000. 12 inch 24 led light tube 60 lumens led light tubes led accent lighting super bright leds. [Daltaoman.com].
Car Interior Accent Lights Autos are one varieties of travel that individuals discover for you to really do the most comfortable currently any days. You will find those who find themselves enthusiastic about vehicles plus enjoy these people and everything its features which enables it to talk forevery preaching about these individuals at the stretch. With the growth of better technology, various new features get show up and already the daily just about all current autos thoroughly designed with those options are noticed coming into the markets. Using the exorbitant selling price are available your terrific capabilities, each one getting among the kind.
Similar to a household, a car can be a type of investment strategies that will you'll make with your lifetime. Consequently it is vital that you do your account throughout negotiating your car loan package you will be consuming to get the investment selling price the small sum of attainable, down to ones relaxation level. Have a seat using the sales agents and keep ailments and condition before these people, tell them precisely what you can pay for and the way a great deal you actually will be able to fork out, and also keeping that in mind get the necessary steps following that on. End up being certainly obvious from the beginning with cash associated issues.
One of the benefits involving online is usually that you could get your vehicle you have been looking during an amazing reduced pace as compared to what you should receive within the showrooms. The following online as well offers you the chance of avoiding the actual annoying sales those that you should bargain with. So you may notice made by this direction it is utterly any win-win scenario that you simply coping listed here, so why wouldn't you enjoy the best standby time with the world wide web to serve this functions that these kinds of awareness such as Car Interior Accent Lights?


Formed in the 1960s, the Northern Chamber Orchestra has gained a formidable reputation performing music from Corelli to Stravinsky and beyond. The members are distinguished chamber musicians who play as principals with other orchestras and as soloists. Concerts are given throughout the northern half of England and the orchestra accompanies the main opera productions at Buxton Festival. The NCO has recorded over thirty CDs with many outstanding reviews: the recording of Stravinsky's The Soldier's Tale was considered by BBC Radio 3's CD Review programme to be the best English version.


text:
Home / Istituzioni culturali
The Library of the House of Literature
The Library of the House of Literature is located in the Borromini complex of the former Oratorio dei Filippini in Piazza dell'Orologio, in the historic centre, and is entirely dedicated to Italian and foreign literature of the twentieth century. Returned to being part of the system of libraries in Rome in 2017, it is the result of the cultural work promoted and carried out by the Department of Cultural Policies and makes use of the collaboration with institutions, universities, public and private cultural associations, both Italian and foreign.
The venue offers various spaces for reading in the galleries, in the two internal rooms and in particular in the cloister which, with its fountain and melangoli grove, represents an ideal place for concentration.
The House of Literature also offers numerous initiatives such as presentations of editorial news, conferences on literary themes, conferences, writing and reading workshops and seminars, bibliographic and documentary exhibitions dedicated to Italian and foreign authors. It also hosts reading clubs, writing workshops and seminars, as well as exhibitions by artists, painters, sculptors and photographers, enhancing intercultural communication.
Inside the library is the collection of the writer, literary critic and playwright Enzo Siciliano, consisting of about 20,000 volumes, monographs, periodicals and old books available to users for consultation only. The core of the works of Italian literature contains many first editions by authors of the twentieth century.
Photo credits: courtesy of The Library of the House of Literature official site
Palazzo dei Filippini (Convento and Oratorio dei Filippini)
Navona Square
The most iconic square of Baroque Rome
POINT (12.468434 41.899141)
For opening times, info and event updates, consult the > official website
casadelleletterature@bibliotechediroma.it
www.facebook.com/casa.letterature
https://twitter.com/BibliotecheRoma - https://twitter.com/bibliotu
www.bibliotechediroma.it/opac/library/biblioteca-casa-delle-letterature/RMBA7
www.youtube.com/channel/UCU_t_e3nQGzM63Uktt2Wcfw
Archivio multimediale
Sala convegni
Sala espositiva
Biblioteca Casa delle Letterature, Piazza dell'Orologio, 3
Piazza dell'Orologio, 3
41° 53' 56.9076" N, 12° 28' 6.3624" E
To find out about all accessibility services, visit the Rome accessible section.
Node Json Map Block
Choose events and services nearby


Enjoy Britt Brandon's Books? Tell us Why and Win a Free Book!
Welcome to Britt Brandon's BookGorilla Author Page! We're working hard to bring you the best deals ever on Britt Brandon's Kindle books. If you're a Britt Brandon fan, we'd love to have your help in making this author page special. Tell us, in no more than 150 words, why you think Britt Brandon is great. Use this webform to send us your submission, and if we select yours we'll offer you one of Britt Brandon's books or a Kindle Gift Card as a free gift! Thanks!
Please enter your email address and click the Follow button to follow Britt Brandon!
The UltimateFitMom.com's Top 100 Tips and Tricks to Lose Weight, Feel Great, Love Yourself, and Live Healthy!


Wedding venues in Glasgow perfectly capture both the urban city feel of modern Glasgow and the tradition and history of some of the most fantastic wedding venues in Scotland.
The city centre is always a buzz, and here you can find modern and stylish hotel wedding venues, like the luxurious Corinthian Club. In the rural countryside, you can find stunning country manors and hotels, such as the stunning Sherbrooke Castle Hotel. Glasgow is the perfect setting for a wedding due to its easy accessibility, rich culture and the genuine hospitality of its' inhabitants who aim only to make your day perfect. No matter what wedding you desire, use our state of the art search engine to refine your search criteria!
At House for an Art Lover, we are passionate about weddings. This Charles Rennie Mackintosh inspired house, set in the grounds of Bellahouston Park is the perfect setting for your beautiful day.
Say I do to the perfect wedding venue... We aim to help make your arrangements as stress free as possible and ensure your Wedding Day is perfect in every way.
Scotland's answer to Downton Abbey!
This award winning 5 star hotel boasts a wonderful elegant setting for your very special wedding day.
Òran Mór is a unique venue situated in the heart of the West End of Glasgow, offering stunning facilities for a wedding ceremony or civil partnership ceremony.
The Busby Hotel is a gorgeous chic venue near Glasgow that aims to make your wedding day a flawless experience. Expect a breathtaking ceremony, lavish accommodation and fantastic service here.