You will be presented with the title and body text of a computer science research paper. Please write a summary of the work that would be informative for a peer reviewer assessing its quality. Your summary should be 5 sentences long. In your response, include only the summary and no additional text.

Paper title:
On the Pitfalls of Analyzing Individual Neurons in Language Models

Paper body:
1 INTRODUCTION . Many studies attempt to interpret language models by predicting different linguistic properties from word representations , an approach called probing classifiers ( Adi et al. , 2017 ; Conneau et al. , 2018 , inter alia ) . A growing body of work focuses on individual neurons within the representation , attempting to show in which neurons some information is encoded , and whether it is localized ( concentrated in a small set of neurons ) or dispersed . Such knowledge may allow us to control the model ’ s output ( Bau et al. , 2019 ) , to reduce the number of parameters in the model ( Voita et al. , 2019 ; Sajjad et al. , 2020 ) , and to gain a general scientific knowledge of the model . The common methodology is to train a probe to predict some linguistic attribute from a representation , and to use it , in different ways , to rank the neurons of the representation according to their importance for the attribute in question . The same probe is then used to predict the attribute , but using only the k-highest ranked neurons from the obtained ranking , and the probe ’ s accuracy in this scenario is considered as a measure of the ranking ’ s quality ( Dalvi et al. , 2019 ; Torroba Hennigen et al. , 2020 ; Durrani et al. , 2020 ) . We see this framework as exhibiting Pitfall I : Two distinct factors are conflated—the probe ’ s classification quality and the quality of the ranking it produces . A good classifier may provide good results even if its ranking is bad , and an optimal ranking may cause an average classifier to provide better results than a good classifier that is given a bad ranking . Another shortcoming of the current methodology , which we mark as Pitfall II , is the focus on encoded information , regardless of whether it is actually used by the model in its language modeling task . A few studies ( Elazar et al. , 2021 ; Feder et al. , 2020 ) have considered this question , and shown that encoded information is not necessarily being used for language modeling , but these do not look at individual neurons . We argue that in order to evaluate a ranking , one should also examine if , and how , the k-highest ranked neurons are used by the model for the attribute in question , meaning that modifying them would change the model ’ s prediction—but with respect to that attribute only . This would allow some control over the model ’ s output , and grant us parameter-level explanations of the model ’ s decisions . In this work , we analyze three neuron ranking methods . Since the ranking space is too large ( 768 ! in BERT ’ s case ) , these methods provide approximations to the problem and are non-optimal . Two of these methods—LINEAR ( Dalvi et al. , 2019 ) and GAUSSIAN ( Torroba Hennigen et al. , 2020 ) —rely on an external probe to obtain a ranking : the first makes use of the internal weights of a linear probe , while the second considers the performance of a decomposable generative probe . The third is a simple ranking method we propose , PROBELESS , which ranks neurons according to the difference in their values across labels , and thus can be derived directly from the data , with no probing involved . 1Our code is available at : < ANONYMIZED > We experiment with disentangling probe quality and ranking quality , by using a probe from one method with a ranking from another method , and comparing the different probe–ranking combinations . We expose the problematic nature of the current methodology ( Pitfall I ) , by showing that in some cases , a suitable probe which is given an intentionally bad ranking , or a random one , provides higher accuracy than another which is given its allegedly optimal ranking . We find that while the GAUSSIAN method generally provides higher accuracy , its probe ’ s selectivity ( Hewitt & Liang , 2019 ) is lower , implying that it performs the probing task by memorizing , which improves probing quality but not necessarily ranking quality . We further find that GAUSSIAN provides the best ranking for small sets of neurons , while LINEAR provides a better ranking for large sets . We then turn to analyzing which ranking selects neurons that are used by the model , by applying interventions on the representation : we modify subsets of neurons from each ranking and measure— using a novel metric we introduce—the effect on language modeling w.r.t to the property in question . We highlight the need to focus on used information ( Pitfall II ) : even though PROBELESS does not excel in the probing scenario , it selects neurons that are used by the model , more so than the two probing-based rankings . We find that there is an overlap between encoded information and used information , but they are not the same , and argue that more attention should be given to the latter . We primarily experiment with the M-BERT model ( Devlin et al. , 2019 ) on 9 languages and 13 morphological attributes , from the Universal Dependencies dataset ( Zeman et al. , 2020 ) . We also experiment with XLM-R ( Conneau et al. , 2020 ) , and find that most of our results are similar between the models , with a few differences which we discuss . Our experiments reveal the following insights : • We show the need to separate between probing quality and ranking quality , via cases where intentionally poor rankings provide better accuracy than good rankings , due to probing weaknesses . • We present a new ranking method that is free of any probes , and tends to prefer neurons that are being used by the model , more so than existing probing-based rankings . • We show that there is an overlap between encoded information and used information , but they are not the same . 2 NEURON RANKINGS AND DATA . We begin by introducing some notation . Denote the word representation space as H ⊆ Rd and an auxiliary task as a function F : H → Z , for some task labels Z ( e.g. , part-of-speech labels ) . Given a word representation h ∈ H and some subset of neurons S ⊆ { 1 , ... , d } , we use hS to denote the subvector of h in dimensions S. For some auxiliary task F and k ∈ N , we search for an optimal subset S∗ such that |S∗| = k and hS∗ contains more information regarding F than any other subvector hS′ , |S′| = k. For the search task , we define neuron-ranking as a permutation Π ( d ) on { 1 , ... , d } and consider the subset Π ( d ) [ k ] = { Π ( d ) 1 , ... , Π ( d ) k } . One may wish to find an optimal ranking Π∗ ( d ) such that ∀k , Π∗ ( d ) [ k ] is the optimal subset with respect to F . However , finding an optimal ranking , or even an optimal subset , is NP-hard ( Binshtok et al. , 2007 ) . Thus , we focus on several methods to produce rankings , which provide approximations to the problem , and compare them . 2.1 RANKINGS . The ranking methods we compare include two rankings obtained from prior probing-based neuronranking methods , and a novel ranking we propose , based on data statistics rather than probing . LINEAR The first method , henceforth LINEAR ( named linguistic correlation analysis in Dalvi et al . 2019 ) ,2 trains a linear classifier on the representations to learn the task F . Then , it uses the trained classifier ’ s weights to rank the neurons according to their importance for F . Intuitively , neurons with a higher magnitude of absolute weights should be more important , or contain more relevant information , for solving the task . Dalvi et al . ( 2019 ) showed that their method identifies important neurons through probing and ablation studies , and found that while the information is distributed across neurons , the distribution is not uniform , meaning it is skewed towards the top-ranked neurons . In this work , we use a slightly modified version of the suggested approach ( Appendix A.1 ) . GAUSSIAN The second method , henceforth GAUSSIAN ( Torroba Hennigen et al. , 2020 ) , trains a generative classifier on the task F , based on the assumption that each dimension in { 1 , ... , d } 2A small enhancement to the algorithm was presented in Durrani et al . ( 2020 ) . is Gaussian-distributed . Then , it makes use of the decomposability of the multivariate Gaussian distribution to greedily select the most informative neuron , according to the classifier ’ s performance , at every iteration . This way we obtain a full neuron ranking after training only once , while applying this greedy method to LINEAR would require retraining the probe d ! times , which is clearly infeasible . Torroba Hennigen et al . ( 2020 ) found that most of the tasks can be solved using a low number of neurons , but also noted that their classifier is limited due to the Gaussian distribution assumption . PROBELESS The third neuron-ranking method we experiment with is based purely on the representations , with no probing involved , making it free of probing limitations ( Belinkov , 2021 ) that might affect ranking quality . For every attribute label z ∈ Z , we calculate q ( z ) , the mean vector of all representations of words that possess the attribute and the value z . Then , we calculate the element-wise difference between the mean vectors , r = ∑ z , z′∈Z |q ( z ) − q ( z′ ) | , r ∈ Rd ( 1 ) and obtain a ranking by arg-sorting r , i.e. , the first neuron in the ranking corresponds to the highest value in r. For binary-labeled attributes , this is simply the difference in means . In the general case , PROBELESS assigns high values to neurons that are most sensitive to a given attribute . 2.2 DATA AND MODELS . Throughout our work , we follow the experimental setting of Torroba Hennigen et al . ( 2020 ) : we map the UD treebanks ( Zeman et al. , 2020 ) to the UniMorph schema ( Kirov et al. , 2018 ) using the mapping by McCarthy et al . ( 2018 ) . We select a subset of the languages used by Torroba Hennigen et al . ( 2020 ) : Arabic , Bulgarian , English , Finnish , French , Hindi , Russian , Spanish and Turkish , to keep linguistic diversity . The tasks we experiment with are predictions of morphological attributes from these languages . Full data details are provided in Torroba Hennigen et al . ( 2020 ) and further data preparation steps are detailed in Appendix A.2 . We process each sentence in pre-trained M-BERT and XLM-R ( unless stated otherwise , all results are with M-BERT ) , and take word representations from layers 2 , 7 and 12 of each model , to see if there are different patterns in the beginning , middle and end of the models . We end up with a total of 156 different configs ( language × attribute × layer ) to test for each model . For words that are split during tokenization , we define their final representation to be the average over their sub-token representations . Thus , each word has one representation for each layer , of dimension d = 768 . We do not mask any words throughout our work .

5-sentence paper summary:
The paper revisits two methods, Linear and Gaussian, as described in the literature, to probe language models based on individual neurons. The paper suggests that these methods contain two limitations - 1. by ranking the neurons on a linguistic task, the methods conflate between ranking quality and the probe's classification quality; 2. the probing methods do not take into account whether the individual neurons are at all used by the model in the downstream linguistic tasks. Finally, the paper presents a new probing method, which does not rely on training a classifier, and shows that this simple method is able to discern among the neurons better than the preceding two methods in the literature.