text:
\section{Relation types list}
\label{sec:relation_types_list}
For \texttt{ConceptNet}, we start with grouping the 31 original \texttt{ConceptNet}\xspace relation types into 17 clusters following \citep{Wang2020ConnectingTD} (Table \ref{tab:17_cn_rel_types} left).
We further group the 31 relation types into 7 by following \citep{liu_2004_conceptnet}, which grouping relation types into 7 categories, including \{things, spatial, events, causal, affective, functional, agents\} (Table \ref{tab:17_cn_rel_types} right).

\begin{table*}

    \footnotesize
    \begin{subfigure}[T]{0.6\textwidth}
    \begin{tabular}{ cp{3cm} | cp{3cm} }
    \toprule
    \multicolumn{4}{c}{\bf 17 relation types} \\
	\midrule
	1 &atlocation, locatednear & 10& usedfor \\\hline
	2 &capableof & 11 &receivesaction \\\hline
	3 &createdby & 12& madeof \\\hline
	4 &desires &  13 &partof, hasa\\\hline
	5 &hascontext & 14 &notdesires \\\hline
	6 &hasproperty & 15&notcapableof \\\hline
	7 &antonym, distinctfrom & 16 &isa, instanceof, definedas \\\hline
	8& relatedto, similarto, synonym& 17& causes, causesdesire,  motivatedbygoal \\\hline
	9& hassubevent, hasfirstsubevent, haslastsubevent,  hasprerequisite, entails, mannerof\\
	\bottomrule
    \end{tabular}
    \end{subfigure}%
    \begin{subfigure}[T]{0.4\textwidth}
    \begin{tabular}{cp{5cm}}
    \toprule
    \multicolumn{2}{c}{\bf 7 relation types}  \\\midrule
       1& capableof\\\hline
       2& usedfor, receivesaction \\\hline
    3&    atlocation, locatednear, hascontext, similarto \\\hline
      4&  causes, causesdesire, motivatedbygoal, desires \\\hline
    5&    antonym, distinctfrom, notcapableof, notdesires \\\hline
     6&   isa, hasproperty, madeof, partof, definedas, instanceof, hasa, createdby, relatedto, synonym \\\hline
      7&  hassubevent, hasfirstsubevent, haslastsubevent, hasprerequisite, entails, mannerof \\\bottomrule
    \end{tabular}

    \end{subfigure}

    \caption{Conflation of relation types in \texttt{ConceptNet}\xspace to either 17 (left) or 7 (right) coarser grained groups.}
    \label{tab:17_cn_rel_types}
\end{table*}

\section{Negation List}
We use a list of negation markers to identify negated words and phrases.
\label{sec:negation_list}
 The list of negation markers is:  (`no', `not', `none', `nor', `no one', `nobody', `nothing', `neither', `nowhere', `never', `hardly', `barely', `scarcely', `non', `without', `fail', `cannot', `cant', `no longer', `dont', `wont').
 Some negated triples sampled from \texttt{ConceptNet}\xspace ans \texttt{SWOW}\xspace are presented in Table~\ref{tab:negated_triples_cn_sw}.

\section{Hyper-parameters}
\label{sec:hyper_parameters}
We use the cross-entropy as the loss function with RAdam \citep{Liu2020ORadam} as optimizer to train all the models. We use as GELU \citep{hendrycks2016gelu} as the activation function. We report most important hyper-parameters in Table~\ref{tab:overall_hyperparameters}.

\section{Re-Production Results}
\label{sec:re_production_results}
For all KG-augmented models, we use the implementation from previous work \citep{Wang2020ConnectingTD}. \footnote{\url{https://github.com/INK-USC/MHGRN}} Table~\ref{tab:reimplementation_roberta_csqa_obqa} compares our re-run results to the original numbers reported in the respective papers. All our reproduced scores are comparable to or better than reported numbers. All of our experiments are run on single GPU of NVIDIA V100 16G.

\input{tables/negated_triples_cn_sw}
\input{tables/overall_hyperparameters}
\input{tables/reimplementation_roberta_csqa_obqa}

\section{Full list of shortest paths }
\label{sec:full_list_of_shortest_paths}
Table~\ref{tab:shortest_path_full_list_cn_sw_183} presents all directed connected paths shown in Table~\ref{fig:narrative_kg_graphs_183} and their corresponding shortest paths retrieved \texttt{ConceptNet}\xspace and \texttt{SWOW}\xspace.

\begin{table*}[!t]
    \centering
    \footnotesize
    \begin{tabular}{llll}
	\toprule
	 & MCScript & \texttt{ConceptNet}\xspace & \texttt{SWOW}\xspace \\
	\midrule
	1 & (purchase, seed) & (purchase, sale, full, seed) & (purchase, need, seed) \\
	2 & (purchase, fertilizer) & (purchase, chain, garage, fertilizer) & (purchase, product, fertilizer) \\
	3 & (hole, cover) & (hole, opening, cover) & (hole, band, cover) \\
	4 & (flower, appear) & (flower, visit, appear) & (flower, become, appear) \\
	5 & (flower, pollinate) & (flower, pollen, pollinate) & (flower, bee, pollinate) \\
	6 & (flower, start) & (flower, open, start) & (flower, green, start) \\
	7 & (flower, bee) & (flower, bee) & (flower, bee) \\
	8 & (grow, continue) & (grow, carry, continue) & (grow, extend, continue) \\
	9 & (grow, vegetable) & (grow, fruit, vegetable) & (grow, vegetable) \\
	10 & (grow, plant) & (grow, plant) & (grow, plant) \\
	11 & (grow, weed) & (grow, field, weed) & (grow, weed) \\
	12 & (continue, pick) & (continue, carry, pick) & (continue, stick, pick) \\
	13 & (pollinate, bee) & (pollinate, pollen, bee) & (pollinate, bee) \\
	14 & (ripen, begin) & (ripen, change, action, begin) & (ripen, blossom, begin) \\
	15 & (garden, water) & (garden, earth, water) & (garden, water) \\
	16 & (garden, remove) & (garden, cricket, remove) & (garden, wart, remove) \\
	17 & (weed, remove) & (weed, remove) & (weed, remove) \\
	18 & (small hole, dig) & (small hole, mouse, hole, dig) & - \\
	19 & (bee, arrive) & (bee, branch, leave, arrive) & (bee, fly, arrive) \\
	20 & (day, make) & (day, clear, make) & (day, present, make) \\
	21 & (few seed, place) & - & - \\
	22 & (gardening tool, purchase) & (gardening tool, tool, lever, purchase) & - \\
	\bottomrule
\end{tabular}
    \caption{ Full list of paths from \texttt{ConceptNet}\xspace and \texttt{SWOW}\xspace for example shown in Figure~\ref{fig:narrative_kg_graphs_183}. - indicates path are not retrieved in the target KG.}.
    \label{tab:shortest_path_full_list_cn_sw_183}
\end{table*}

\section{Experiment 2: Coverage of Commonsense Knowledge}
\label{sec:experiment_two}
This section probes \texttt{ConceptNet}\xspace and \texttt{SWOW}\xspace specifically for (situational) {{\it commonsense} knowledge}. Daily activities such as {\it doing the laundry} or {\it visiting the doctor} involve a wealth of general knowledge touching on causal, temporal, physical, or social knowledge which is rarely explicitly stated~\cite{Mostafazadeh2016ACA,rashkin-etal-2018-modeling,ostermann-etal-2018-mcscript,ostermann-etal-2019-mcscript2}.
We leverage the MCScript2.0 data set~\citep{ostermann-etal-2019-mcscript2}, a large collection of {\it explicit} descriptions of everyday scenarios, and investigate whether \texttt{ConceptNet}\xspace and \texttt{SWOW}\xspace encode the commonsense knowledge underlying these situations. In particular, we test~whether the knowledge graph structure underlying MCScript scenarios is retained in \texttt{ConceptNet}\xspace and \texttt{SWOW}\xspace. Figure~\ref{fig:narrative_kg_graphs_183} shows an example of a scenario (a), derived graph representation (b), and a subset of corresponding associations in \texttt{ConceptNet}\xspace (c) and \texttt{SWOW}\xspace (d).

\subsection{Method}
\label{subsec:narrative_commonsense_knowledge_comparison}
 \input{figures/shortest_path_length_hist_cn_sw.tex}

\paragraph{MCScript2.0} is a collection of 3,487 short narrative descriptions covering 200 every-day scenarios of varying complexity (e.g., {\it cleaning the floor} vs {\it growing vegetables})~\cite{ostermann-etal-2019-mcscript2}. The descriptions were crowd-sourced, and authors were instructed to describe the underlying scenarios ``as if talking to a child''~\cite{ostermann-etal-2018-mcscript}. Thus by design MCScript narratives spell out commonsense knowledge more explicitly than most text corpora.
Following prior work on modelling narrative scripts, we posit that narrative chains are fundamentally characterized in terms of their events and participants~\cite{chambers2009unsupervised,frermann2014hierarchical}.
We recover this information using semantic role labelling (SRL), and identify predicates, ARG0s and ARG1s in each narrative. We use the resulting set of spans and their relations to transform each narrative (Figure~\ref{fig:narrative_kg_graphs_183}a) into a {\it script graph} (Figure~\ref{fig:narrative_kg_graphs_183}b). Spans constitute the nodes of the script graph,\footnote{Pronouns and stopwords are excluded. We use the SRL model of~\citet{Shi2019SimpleBM} implemented in AllenNLP.} and edges correspond to (undirected) SRL dependencies.

\paragraph{Graph mapping}
We assume that predicates-argument relations in detailed descriptions of everyday events (e.g., `water' and `garden') are instances of common sense knowledge, and should hence be directly encoded in a KG.
We posit that the shorter the paths, the more directly the necessary scenario-specific commonsense knowledge is encoded in the target KG. E.g.,  we find a direct 1-hop connection between `water' and `garden' in \texttt{SWOW}\xspace (Figure~\ref{fig:narrative_kg_graphs_183}d), while the path between the concepts in \texttt{ConceptNet}\xspace is less direct (2-hops;  Figure~\ref{fig:narrative_kg_graphs_183}c).

More technically, we project the MCScript graphs onto \texttt{SWOW}\xspace and \texttt{ConceptNet}\xspace as follows. For all directly connected node pairs in a MCScript graph, we identify their corresponding concepts in \texttt{SWOW}\xspace and \texttt{ConceptNet}\xspace, respectively, by exact string match. In our current study we are primarily interested in {\it whether} (not how) node pairs from the MCScript graph exist in the target KGs. {We therefore treat all graphs as undirected, and retrieve the shortest path between the two nodes in the target KGs.}
Figure~\ref{fig:narrative_kg_graphs_183}c and \ref{fig:narrative_kg_graphs_183}d show a subset of the shortest paths retrieved from \texttt{ConceptNet}\xspace and \texttt{SWOW}\xspace, respectively, for the MCScript graph in \ref{fig:narrative_kg_graphs_183}b (bolded nodes). See Appendix ~\ref{sec:full_list_of_shortest_paths} for the full list of retrieved paths from \texttt{ConceptNet}\xspace and \texttt{SWOW}\xspace.

\subsection{Results}
Figure~\ref{fig:shortest_path} presents the distribution over shortest path lengths, averaged over the full MCScript data set. We observe that paths, on average, are shorter in \texttt{SWOW}\xspace compared to \texttt{ConceptNet}\xspace, suggesting that the situational commonsense associations in MCScript are more directly encoded in \texttt{SWOW}\xspace. The example shortest paths shown in Figure~\ref{fig:narrative_kg_graphs_183}c (\texttt{ConceptNet}\xspace) and d (\texttt{SWOW}\xspace) further illustrate the associations in the two commonsense resources. The associations imposed in paths of length ${>}1$ are meaningful across the board, but differ across the KGs: for example, the connection between `bee' and `arrive' is further elaborated in \texttt{ConceptNet}\xspace by explaining that in order to arrive, the bee needs to leave (from a plausible location `branch'); \texttt{SWOW}\xspace on the other hand imputes information on the mode of travel (`flying').

As a first exploration into evaluating the extent of commonsense knowledge in commonsense KGs, our method has several weaknesses to be addressed in future work: the string mapping from text to commonsense knowledge\xspace could be replaced with more flexible, embedding-based methods; the script graphs themselves could be improved and enriched with more semantic roles, or higher-level narrative structure as captured for instance in Rhetorical Structure Theory~\cite{taboada2006rhetorical}. Similarly, the mapped graphs could incorporate edge directions and/or labels. Ultimately, the mapped paths (most interestingly those mapped to paths of length $>1$) will need to be validated and interpreted by human annotators. We believe that leveraging explicit human-created commonsense data sets, like MCScript2.0, opens interesting avenues to understand the commonsense knowledge present in word associations.

\section{Experiment 1: Intrinsic Comparisons}
\label{sec:experiment_one}
A knowledge graph  consists of a set of nodes $\mathcal{N}$ and edges $\mathcal{E}$, comprising triples $\langle e, r, e'\rangle$ to denote a directed edge from head node $e$ to tail node $e'$ labelled with relation $r \in \mathcal{R}$. We denote $\mathcal{E}(e)$ as the incoming and outgoing edge set for node $e$, $\mathcal{E}(r)$ as the set of edges with relation~$r$, and $|\cdot|$ as the size of a set. Here, we consider the specific knowledge graphs \texttt{ConceptNet}\xspace and \texttt{SWOW}\xspace, and begin by comparing the intrinsic properties: their typology and content encoded.

\subsection{Knowledge Graph Structure}
\label{ssec:knowleddge_graph_structure}
\texttt{ConceptNet}\xspace is a substantially larger graph than \texttt{SWOW}\xspace, with about eight times as many nodes and 1.9$\times$ as many edges (cf.,~Table \ref{tab:statitics_cn_rel47_sw}). We compare sparsity in terms of (1)~{\it graph density},
\begin{equation*}
    \frac{|\mathcal{E}|}{|\mathcal{N}|(|\mathcal{N}|-1)},
\end{equation*}
and (2)~{\it node degree} as the average total of incoming and outgoing edges~\citep{Malaviya2019ExploitingSA}. Table~\ref{tab:statitics_cn_rel47_sw} shows that \texttt{SWOW}\xspace has 39$\times$ the density and 4$\times$ the average node degree of \texttt{ConceptNet}\xspace: Even though \texttt{SWOW}\xspace is smaller than \texttt{ConceptNet}\xspace, it is substantially more densely connected.

\paragraph{Node Distribution.}
To better understand the distribution of nodes in the KGs, we  measure node diversity via the entropy of the node distribution~($H_N$; \newcite{pujara-etal-2017-sparsity}):
\begin{equation*}
\begin{aligned}
    H_N &=\sum_{e \in N}-P(e) \log P(e),\\
\end{aligned}
\end{equation*}
where $P(e) = |\mathcal{E}(e)| / |\mathcal{E}| $ is the fraction of edges incident on the node.

Higher $H_N$ indicates a more uniform node distribution where many nodes are connected, whereas lower entropy suggests a skewed distribution, where few nodes are highly connected. Table~\ref{tab:statitics_cn_rel47_sw} shows a lower $H_N$ for \texttt{SWOW}\xspace, i.e., nodes are less uniformly connected. This is because \texttt{SWOW}\xspace is by construction more structured than \texttt{ConceptNet}\xspace: its  12K \textit{cue} nodes are densely connected to a much larger number of (sparsely connected) \textit{response} nodes.

\paragraph{Node and Edge Overlap.}
\label{ssec:shared_knowledge}

How large is the overlap between \texttt{ConceptNet}\xspace and \texttt{SWOW}\xspace? We quantify the overlap of individual nodes in the two KGs, based on exact string match.\footnote
String matching is arguably a simplistic way of matching concepts across two KGs. Advanced methods of concept resolution could leverage embedding methods. We leave this interesting direction for future work.}
We find 58\%  (71K) of the nodes in the smaller \texttt{SWOW}\xspace are present in \texttt{ConceptNet}\xspace (conversely, 7\% of the nodes in \texttt{ConceptNet}\xspace are present in \texttt{SWOW}\xspace). Over 40\% of the concepts in \texttt{SWOW}\xspace are not present in \texttt{ConceptNet}\xspace, which is perhaps expected given their very distinct methods of construction, but motivates further in-depth comparison (Section~\ref{subsec:edge_type_comparison}). Moving on to edge overlap, which we measure over undirected head-tail pairs,%
\footnote{Edge comparison ignores direction as many relations can naturally be inverted, e.g., `part of' and `has part'. Consequently linking concepts in either direction is considered to be correct.}
we find that 6\% of edges in \texttt{ConceptNet}\xspace are present in \texttt{SWOW}\xspace, and 15\% of the edges in \texttt{SWOW}\xspace are present in \texttt{ConceptNet}\xspace.
This low overlap demonstrates that human associations indeed elicit connections among words missed in the large database \texttt{ConceptNet}\xspace.
For the 71K overlapping nodes, we further find that 691K connections exist in \texttt{ConceptNet}\xspace, and 1.5M connections in \texttt{SWOW}\xspace, covering 95\% of all \texttt{SWOW}\xspace edges. This again suggests that \texttt{SWOW}\xspace is more comprehensive than \texttt{ConceptNet}\xspace.

\input{figures/tag_node_pos_phrase_cn_sw_ybar.tex}
\subsection{Knowledge Graph Content}
\label{subsec:edge_type_comparison}
Having established the structural characteristics of \texttt{ConceptNet}\xspace and \texttt{SWOW}\xspace, we will now focus on their respective encoded knowledge.

\subsubsection{Conceptual Content}

Nodes in \texttt{ConceptNet}\xspace and \texttt{SWOW}\xspace express concepts as words or short phrases. We compare: (1) the distributions of the syntactic categories for concepts over the two knowledge bases; (2) the occurrences of concepts in two KGs in large corpus.

We use a constituency parser to predict the syntactic phrase or part of speech (POS) tag for a concept string.\footnote{We use the parser of~\newcite{Kitaev-2018-SelfAttentive} as implemented in Spacy.}
The relative prevalence of the 10 most frequent syntactic types is shown in Figure~\ref{fig:tag_node_pos_phrase_cn_sw}. While the overall distribution is similar in both KGs, two patterns emerge. First, even though both KGs are dominated by nominal nodes, \texttt{SWOW}\xspace's distribution over POS types is less skewed, suggesting concepts are more diverse. Secondly, the proportion of phrasal concepts compared to single-word concepts tends to be higher in \texttt{SWOW}\xspace compared to \texttt{ConceptNet}\xspace.

Next we examine the corpus frequency of concepts in \texttt{ConceptNet}\xspace and \texttt{SWOW}\xspace using the Google n-gram corpus.\footnote{\url{https://books.google.com/ngrams}}
Many \texttt{ConceptNet}\xspace concepts were not present in this large corpus (24\%), versus 1.5\% for \texttt{SWOW}\xspace.
Of those concepts that could be found, concepts in \texttt{SWOW}\xspace are on average $7\times$ more common than those in \texttt{ConceptNet}\xspace.
We conclude that \texttt{SWOW}\xspace concepts are generally common, while \texttt{ConceptNet}\xspace includes more obscure concepts.

\subsubsection{Relational Content}
We cannot directly compare relation distributions between \texttt{ConceptNet}\xspace and \texttt{SWOW}\xspace because \texttt{SWOW}\xspace does not have labelled relations. Instead, we inspect the intersection of the two graphs (\texttt{CN$\cap$SW}\xspace) as all head-tail pairs that are shared between the graphs ($|\mathcal{E}_{\text{\texttt{CN$\cap$SW}\xspace}}|$=190K), labelled with their \texttt{ConceptNet}\xspace relations. We use \texttt{CN$\cap$SW}\xspace as a proxy of the relations in \texttt{SWOW}\xspace.
For each relation type $r$, we compare its relative frequency in the full \texttt{ConceptNet}\xspace ($f^r_{\texttt{CN}}$) against its recall in \texttt{CN$\cap$SW}\xspace ($\text{recall}^r$), where
\begin{equation*}
 \begin{aligned}
    f^r_{\text{\texttt{CN}}} &= \frac{|\mathcal{E}_{\texttt{CN}}(r)|}{|\mathcal{E}_{\texttt{CN}}|}, &
    \text{recall}^r &= \frac{|\mathcal{E}_{\texttt{CN$\cap$SW}\xspace}(r)|}{|\mathcal{E}_{\texttt{CN}}(r)|}.
 \end{aligned}
\end{equation*}

\input{figures/relation_distribution3}
Figure~\ref{fig:relation_distribution} plots the correlation between the $f^r_{\texttt{CN}}$, and their recall in \texttt{CN$\cap$SW}\xspace. First, we observe a discenerable correlation between the majority of low- to medium frequency relations in \texttt{ConceptNet}\xspace and their recall in \texttt{CN$\cap$SW}\xspace (\textcolor{orange}{orange box} on bottom left).
These relations cover largely semantic associations pertaining to the appearance, use or situational contexts of concepts. Second, none of the six most frequent relation types in \texttt{ConceptNet}\xspace (right part of the plot), are highly recalled in \texttt{CN$\cap$SW}\xspace. Out of these, relations indicating (near) synonymy retain a medium recall (\textcolor{red}{red box} on middle right), while morphosyntactic relations are less prevalent (\textcolor{blue}{blue box} on bottom right). These relations are prevalent in \texttt{ConceptNet}\xspace, because it is derived to a large part from structured, linguistic resources like Wiktionary.\footnote{74\% of \texttt{ConceptNet}\xspace edges origin from Wiktionary.} Word associations, on the other hand, are known to be dominated by semantic relations~\cite{mollin2009combining}.
Third, the relations with highest recall in \texttt{CN$\cap$SW}\xspace tend to be infrequent in \texttt{ConceptNet}\xspace (\textcolor{green}{green box} on top left).
Two of these relations focus on {\it differences} indicating that humans associate contrasting concepts (such as `hot'$\rightarrow$`not cold'; \citet{Deese1964TheAS,Herbert1970WordAsso}).
We also found that the proportion of negated edges in \texttt{SWOW}\xspace (0.7\%; N=11K) is more than twice the proportion in \texttt{ConceptNet}\xspace (0.3\%; N=11.5K), and that only 4\% of them overlap.\footnote{Negated nodes were identified based on a list of negation markers, cf.,~Appendix~\ref{sec:negation_list}.} Representations of antonyms and negations have traditionally been difficult to infer from text, suggesting that \texttt{SWOW}\xspace could be a valuable complementary knowledge source to fill this gap.

\section*{Acknowledgements}

This work was supported in part by China Scholarship Council (CSC). We thank the reviewers for their valuable comments, and Simon De Deyne for insightful discussions.

\section{Experiment 3: Commonsense QA}
\label{sec:experiment_three}
In this section, we explore the utility of commonsense knowledge in \texttt{ConceptNet}\xspace and \texttt{SWOW}\xspace in commonsense question answering (CQA) tasks. We incorporate the two KGs into representative and competitive CQA models from the recent literature \citep{Wang2020ConnectingTD,Feng2020ScalableMR}, and apply them to three benchmark data sets. We emphasize that the goal of this study is not competing on leaderboards. Current state-of-the-art models leverage very large language models with billions of parameters \citep{Khashabi2020UnifiedQACF}, and often draw on additional external resource such as Wiktionary \citep{Xu2020FusingCI}.
Instead, we explore the utility of \texttt{SWOW}\xspace and \texttt{ConceptNet}\xspace in a selection of representative moderately complex models.

\subsection{Experimental setup}
\label{sec:models}
\input{tables/datasets_table}
\input{tables/cqa_main.tex}
\paragraph{Datasets}
We consider three standard multiple-choice CQA benchmark datasets. \textbf{CommonsenseQA} (CSQA; \newcite{talmor-etal-2019-commonsenseqa}) contains commonsense questions generated by crowd workers on the basis of sub-graphs in \texttt{ConceptNet}\xspace, giving \texttt{ConceptNet}\xspace an inherent advantage over \texttt{SWOW}\xspace. The QA-pairs in this dataset require various commonsense skills, and distractor answers were carefully selected to share semantic associations with the key concepts in a question.
\textbf{OpenBookQA} (OBQA; \newcite{Mihaylov2018CanAS}) consists of question-answer pairs along with paragraphs from elementary-level science books. Following previous work~\citep{Wang2020ConnectingTD,Feng2020ScalableMR}, we disregard the paragraphs, and apply our models to question-answer pairs directly.
Building on our analysis in Section~\ref{sec:experiment_two}, we also apply our models to the \textbf{MCScript2.0} QA benchmark~\citep{ostermann-etal-2019-mcscript2}. Each task consists of a story, and a question paired with two answer options.
We use the in-house data split by \newcite{lin2019kagnetkg} for CSQA and the official splits for the other data sets. Table~\ref{tab:dataset_examples} presents data set statistics, as well as an example from each dataset.

\paragraph{Models}
A typical QA system consists of three modules: (1) a KG encoder which maps a subgraph spanning the concepts in the question/answer to a fixed-dimensional knowledge embedding
$\mathbf{k}$;(2) a text encoder, which maps question $q$ and answer $a$ to a fixed-dimensional language embedding $\mathbf{c}$; and (3) a scoring module which scores each answer option given $\mathbf{c}$ and $\mathbf{k}$, and returns the highest scoring answer as a prediction.

We experiment with two KG encoders: {\sf GconAttn}~\citep{Wang2019ImprovingNL} maps question and answer concepts to pre-trained concept embeddings, and then aligns them using concept-level attention and pooling. GconAttn is a relation-free model, leveraging only mentioned concepts from a KG. To disentangle the impact of relation labels, we also experiment with {\sf Relation Networks (RN)}~\citep{Santoro2017ASN}, which embed concepts using context-aware path-level attention over path embeddings. Each path embedding  encodes the path between some question concept and answer concept. We use one-hop and two-hop paths in our experiments. We refer the reader to the respective papers for full model details, as well as  Appendix~\ref{sec:re_production_results} which details our full parameter settings and reproduction results.
Both KG encoders require {\em node embeddings}. We use RoBERTa-Large \citep{Liu2019RoBERTaAR} to obtain a node embedding matrix, separately for \texttt{ConceptNet}\xspace and \texttt{SWOW}\xspace.
Specifically, for each node $\mathcal{E}_i \in \mathcal{E}$, we feed the sequence of \text{[\texttt{CLS}]  $ + \, \mathcal{E}_i +$ [\texttt{SEP}]} to RoBERTa and use the last layer representation of [\texttt{CLS}] as its embedding.

RN also requires {\em relation embeddings}. For each of our KGs (\texttt{ConceptNet}\xspace, \texttt{SWOW}\xspace), we obtain a separate relation embedding matrix $\mathbf{R}$ with {\sf TransE}.\footnote{Using OpenKE \url{https://github.com/thunlp/OpenKE}. We do not use TransE node embeddings, as they were outperformed by RoBERTa embeddings in preliminary tests.}
Following previous work on CQA~\citep{lin2019kagnetkg,Wang2020ConnectingTD}, we select 31 out of \texttt{ConceptNet}\xspace's 47 relation types which proved helpful for CQA, and merge the remaining relations into 17 types. We use forward- and mutual associations for \texttt{SWOW}\xspace.  Following previous work ~\citep{Malaviya2019ExploitingSA,Wang2020ConnectingTD}, we densify both \texttt{SWOW}\xspace and \texttt{ConceptNet}\xspace by adding for each relation $\mathcal{R}_i$ an additional relation type indicating its reverse $\mathcal{R}_i^{-1}$. For example, a relation $\mathcal{R}_i{=}\text{part\_of}$ would be complemented with $\mathcal{R}_i^{-1}{=}\text{has\_part}$.

We fix the text encoder to {\sf ALBERT-xxlarge-v2}~\citep{lan2020albertal}, which performed competitively in recent work~\citep{Wang2020ConnectingTD}.
All models are run six times and we report results using the best three models, as judged by training loss; this method is used to remove outliers resulting from instability of training.
All results are our own re-runs using the official implementations from \url{https://github.com/INK-USC/MHGRN}, and are largely comparable with those reported in the literature. See Appendices~\ref{sec:hyper_parameters},~\ref{sec:re_production_results} for further details.

\subsection{Results}
\label{sec:experimental_results}

Experimental results in Table~\ref{tab:results_main_table_csqa} show that both knowledge-augmented models outperform the language baseline for all data sets except RN on MCScript2.0. The path-aware RN achieves the best performance for both CSQA and OBQA. All models (including the text-only baseline) show comparative and very high performance on MCScript2.0, suggesting that it is a simpler task compared to the other two. Our models outperform the current state-of-the art on MCScript2.0 by up to 3.24\% (absolute).\footnote{\url{https://coinnlp.github.io/task1.html}} More importantly, models incorporating either of \texttt{ConceptNet}\xspace or \texttt{SWOW}\xspace achieve similar performance across the board. Recall that CSQA is derived from \texttt{ConceptNet}\xspace edges, putting \texttt{SWOW}\xspace at a disadvantage. \texttt{SWOW}\xspace performs best on OBQA which is independent of both KGs. {We measure the significance of differences in performance between the text-only baseline and the RN models on CSQA and OBQA using Student's t-test. We find that the RN models outperform the text-only model significantly ($p{<}0.05$) with both \texttt{ConceptNet}\xspace and \texttt{SWOW}\xspace as underlying KG.\footnote{The only exception is ALBERT vs RN with \texttt{ConceptNet}\xspace on OBQA where~$p{=}0.07$.} There is no significant difference between the \texttt{ConceptNet}\xspace-based and the \texttt{SWOW}\xspace-based RN models ($p{>>}0.05$).}
These results provide initial evidence that \texttt{SWOW}\xspace can be a valuable alternative source of commonsense knowledge to \texttt{ConceptNet}\xspace for downstream NLP tasks.

\input{figures/relation_diversity_csqa_obqa.tex}

The competitive performance of \texttt{SWOW}\xspace may be surprising, particularly with the relation-aware KG encoder RN, because \texttt{ConceptNet}\xspace has access to a rich typed relation inventory while \texttt{SWOW}\xspace does not. We investigate the impact of labelled relation types on CQA for RN, by ablating the number of relation types accessible to \texttt{ConceptNet}\xspace: we grouped the 17 \texttt{ConceptNet}\xspace relation types used in the models above into (a) seven coarse types (plus reverse relations) using the relational ontology of \newcite{liu_2004_conceptnet};\footnote{See Appendix~\ref{sec:relation_types_list} for details of 17 and 7 relation types.} (b) a single generic relation type (plus reverse relation). This version still contains one-hop and two-hop paths as well as reverse relations; and (c) removing all relation information from the model.
Figure~\ref{fig:relation_diversity_csqa_ihtest} shows that merging \texttt{ConceptNet}\xspace relations into broader types improves downstream task performance across data sets, with the best results achieved with one or seven relation types. Our results suggest that augmenting \texttt{SWOW}\xspace with a rich label inventory may not be necessary for it to be used as a common-sense resource in downstream common-sense reasoning models. Nevertheless, labelling \texttt{SWOW}\xspace with cognitively valid relation (or commonsense type) information in order to better understand the types of spontaneous associations humans express is an exciting avenue for future work.

\section{Introduction}
Humans understand and navigate everyday situations with great efficiency using a wealth of shared, basic facts about their social and physical environment -- a resource often called commonsense knowledge \citep{liu_2004_conceptnet}. Advances in artificial intelligence in general, and natural language processing in particular, have led to a surge of interest in its nature: what constitutes commonsense knowledge? And despite this knowledge being rarely explicitly stated in text~\citep{Gordon2012ReportBias}, how can we equip machines with commonsense to enable more general inference~\citep{Davis2015CommonsenseRA}? Recently, large language models \citep{devlin2019bertpo,radford2019language} pre-trained on massive text corpora achieved promising results on commonsense reasoning benchmarks with fine-tuning, however, the lack of interpretability remains a problem. Augmenting them with commonsense knowledge can provide complementary knowledge \citep{Ilievski2021DimensionsOC,Safavi2021RelationalWK} and thus make models more robust and explainable \citep{lin2019kagnetkg}.

\input{figures/camping_subgraph}

Attempts to capture general human knowledge include inventories of machine-readable logical rules~\cite{gordon2017formal} and large, curated databases which have been collected with the specific purpose to reflect either domain general (e.g., \texttt{ConceptNet}\xspace; \citet{liu_2004_conceptnet}) or domain-specific (e.g., ATOMIC; \citet{sap2019atomicaa}) commonsense knowledge.

Word association norms are a third resource of explicit, basic human knowledge: in a typical study, human participants are presented with a {\it cue} word (e.g., `camping') and asked to produce one or more words that spontaneously come to mind (e.g., `hike', `nature', or `trail'). The resulting data sets of word associations have been used to explore the mental representations and connections underlying human language and reasoning \citep{Nelson2004TheUO,Fitzpatrick2006HabitsAR}, and have been shown to contain information beyond that present in text corpora \citep{de-deyne-etal-2016-predicting}. Word association studies have been scaled to thousands of cue words, tens of thousands of participants, and several languages, and hence provide a way of collecting diverse and unbiased representations. However, the extent to which they capture {\it commonsense} knowledge, and their utility for downstream applications have not yet been examined.

Prior work has probed the extent to which word associations contain lexical knowledge~\cite{de-deyne-etal-2016-predicting,Deyne2019TheW}, has systematically compared the relational coverage and overlap of curated knowledge bases~\cite{ilievski2020consolidating} (not including word associations data), and has investigated how much relational commonsense knowledge is present in language models~\cite{a-rodriguez-merlo-2020-word,vankrunkelsven2018predicting,da2019cracking}. We contribute to this line of research with an in-depth comparison of a dedicated commonsense knowledge base, and resources derived from spontaneous human-generated word associations. We systematically compare the most comprehensive, domain general, curated commonsense knowledge base (\texttt{ConceptNet}\xspace) with the largest data set of English word associations (the ``Small World of Words''; \texttt{SWOW}\xspace;~\citet{Deyne2019TheW}). We compare the two resources both in their structure and content, and apply them in downstream reasoning tasks.

Our contribution is important for three reasons. First, comparing explicitly engineered with spontaneously produced knowledge graphs can advance our fundamental understanding of the similarities and differences between the paradigms, and suggest ways to combine them. Second, recent progress in automatic commonsense reasoning largely focused on English and relies heavily on the availability of very large language models. These are, however,
infeasible to train for all but a few high-resource languages.
Finally, recent work has shown that the competitive performance of large language models on commonsense reasoning tasks is at least partially due to spurious correlations in language rather than genuine reasoning abilities; and that they perform close to random once evaluation controls for such confounds~\cite{elazar2021back}. Explicit representations of commonsense knowledge bases are thus have the potential to promote robust and inclusive natural language reasoning models.

In summary, our contributions are:

\begin{enumerate}
 \item We conduct an in-depth comparison of large-scale curated commonsense knowledge bases and word association networks, distilling a number of systematic differences which can inform future theoretical and empirical work.

 \item We analyze how much {\it commonsense} knowledge \texttt{ConceptNet}\xspace and \texttt{SWOW}\xspace encode, leveraging a human-created data set covering explicit situational knowledge. Our results suggest that \texttt{SWOW}\xspace represents this knowledge more directly.

 \item We introduce \texttt{SWOW}\xspace as a commonsense resource for NLP applications and show that it achieves comparable results with \texttt{ConceptNet}\xspace across three commonsense question answering benchmarks.
\end{enumerate}

\section{Background}
\label{sec:graph_comparisons}
\smallskip

\subsection{Commonsense Knowledge Graphs}
Collecting and curating human commonsense knowledge has a rich history in artificial intelligence and related fields, motivated by the grand challenge of equipping AI systems with commonsense knowledge and reasoning ability~ \citep{Davis2015CommonsenseRA,lake_ullman_tenenbaum_gershman_2017}.
Previous efforts have been put on collecting different aspects of commonsense knowledge, resulting in a variety of resources ranging from systems of logical rules \citep{Lenat1990BuildingLK}, over knowledge graphs \citep{liu_2004_conceptnet}, all the way to embedded representations \citep{Malaviya2019ExploitingSA}. Here, we focus on graph representations. Domain-specific examples include ATOMIC \citep{sap2019atomicaa} which focuses on social interactions in events, SenticNet \citep{Erik2020SenticNet6} which encodes sentiment-related commonsense knowledge. Recent work  also attempts to consolidate knowledge from multiple sources in order to improve knowledge coverage and utility~\citep{Ilievski2021CSKGTC, Hwang2020COMETATOMIC2O}. The largest domain-general commonsense knowledge\xspace graph is \texttt{ConceptNet}\xspace, which we will use in this study and describe in detail below.

\input{tables/graph_statistics_comparisons}

\paragraph{\texttt{ConceptNet}\xspace}
All studies in this paper are based on the most recent \texttt{ConceptNet}\xspace v5.6~\citep{Speer2017Conceptnet5.5}. \texttt{ConceptNet}\xspace is a directed graph comprising over 3M nodes (aka concepts). Related concepts are connected with directed edges, which are labelled with one of 47 generic relation types.
Figure~\ref{fig:camping_cn_sw}a shows a small subgraph, centered around the concept `camping'. Nodes are represented as free-text descriptions, which leads to a large node inventory and a sparsely connected graph. We filter out nodes that are not English, lowercase all descriptions and remove punctuation.  Row 1 in Table~\ref{tab:statitics_cn_rel47_sw} shows statistics of the resulting knowledge graph.

\subsection{Word Association Networks}
Human word associations have a long history in psychology and cognitive linguistics~ \citep{Deese1966TheSO,Deyne2019TheW}. Given a {\it cue} word, one or more spontaneous {\it responses} are elicited from study participants, shedding light on their mental associations. The resulting data sets, covering many cues and participants, can subsequently be turned into association {\it networks}, where each node corresponds to either a cue or response. Cues are connected to the responses they elicited through directed edges, which can be weighted (or filtered) by the number of participants who produced a particular response. The primary use-case of word associations has been to gain insights into the mental lexicon. A wealth of studies has shown the utility of word associations for predicting behavioural data including memory, lexical choice and semantic categorization~\cite{Nelson2000WhatIF,de2013associative,borge2010categorizing}, however, we are the first to inspect word association networks as a general commonsense knowledge base. Several association data sets have been collected, varying in coverage and target language (\newcite{Armstrong1973EAT,Nelson2004TheUO,jung2010network,Deyne2013BetterEO}). However, in order to consider this data as a general knowledge resource, it should (a)~cover a large set of diverse {cues}, and (b)~a large number of {responses} which are both diverse and reliable. Recently, the \emph{Small World of Words} project massively scaled word association collection through crowd-sourcing \cite{Deyne2019TheW}. In the remainder of this paper, we  use their English data set (``\texttt{SWOW}\xspace'', described below). The \emph{Small World of Words} project has been scaled to 15 languages, suggesting its potential as a knowledge resource for NLP more generally.

\paragraph{Small World of Words}
\texttt{SWOW}\xspace~\citep{Deyne2019TheW} is a word association network derived from a large collection of crowd-sourced cue-response pairs involving more than 90K participants and 12K cues. Given a cue, participants produced up to three responses. The resulting associations have been compiled into the \texttt{SWOW}\xspace knowledge graph (Figure~\ref{fig:camping_cn_sw}b shows a small excerpt).
\texttt{SWOW}\xspace edges are not labelled with relations. In some of our experiments, we adopt a basic set of two relations using the associative directions, namely {\it forward associations} from a cue to a response, and {\it mutual associations}  for pairs where the reverse is also included in \texttt{SWOW}\xspace. We use the official, pre-processed release of \texttt{SWOW}\xspace.\footnote{\url{https://smallworldofwords.org/en/project/research}} We remove ``NA'' responses, lowercase all node descriptions and remove punctuation. Row 2 in Table~\ref{tab:statitics_cn_rel47_sw} shows statistics of the resulting graph.

In the remainder of this paper, we conduct three experiments to compare \texttt{SWOW}\xspace and \texttt{ConceptNet}\xspace from different dimensions, ranging from the intrinsic graph properties (\S\ref{sec:experiment_one}), to the coverage of encoded commonsense knowledge (\S\ref{sec:experiment_two}), and their utility for  downstream commonsense reasoning tasks (\S\ref{sec:experiment_three}).
\section{Conclusions}

We presented an in-depth analysis of the general and commonsense knowledge encoded in human word association norms (\texttt{SWOW}\xspace), versus a traditional curated commonsense knowledge graph (\texttt{ConceptNet}\xspace). We showed that the two knowledge resources differ systematically in their structure and content. We also showed that \texttt{SWOW}\xspace brings comparable gains to \texttt{ConceptNet}\xspace when applied to three commonsense question answering benchmarks, which is important as word associations are simpler than structured relations, and accordingly can be created more cheaply via crowd-sourcing and without the need for experts. Finally, we showed that \texttt{SWOW}\xspace encodes situational commonsense knowledge as encoded in the human-created MCScript2.0 narratives  more directly than \texttt{ConceptNet}\xspace; and that both KGs impose meaningful additional relations between concepts that were left implicit in the descriptions. There are several directions for future work, most notably extending our framework for characterizing  commonsense knowledge in the two KGs, exploring \texttt{SWOW}\xspace relation types, developing means of consolidating the two knowledge graphs, and exploring downstream utility of KGs in less well-resourced languages than English, where the portability of the word association annotation methodology confers a substantial advantage.
\section{Introduction}

Commonsense reasoning is
and association
The ability to
Given an commonsense question
people can easily associate key concepts in a question with the correct answer
For example, if we observe (Fig 1), we can easily

Teaching machines to
Commonsense knowledge is vital for natural language understanding.
Many tasks have shown that
question answering,
dialogue systems,
story understanding,
require commonsense reasoning about the world
To answer a question like "" require a model to draw on world knowledge like "".

Commonsense knowledge in ConceptNet are usually collected in three ways, including hand-crafted by experts, automatically extracted from the Web, and games with a purpose.

Existing models use commonsense knowledge graph as additional source to assist downstream tasks.
Nodes and relations in a knowledge graph are represented as embeddings and are incorporated to question-answer models.
However, train high quality knowledge graph embeddings models often require the graph is densely connected to learn better representations.
ConceptNet is a giant semantic network containing commonsense facts.
It has been shown that it is  useful for improving accuracy of many NLP tasks.
However, it is sparse and
Knowledge in ConceptNet is collected from three ways: hand-crafted by experts, automatically extracted from the Web, and games with a purpose. which is either time and money consuming or hard to capture the unstated commonsense.

Creating a massive commonsense knowledge graph like ConceptNet in other languages are unrealistic.

Another manner of
One knowledge source of ConceptNet is word associations from purposes with a game, like Verbosity,
collecting commonsense associations from a large auidence of ordinary humans by playing games.
Relations associate two words are categoried by different sentence templates,
And property norms
Another form of association is free word associations, revealing how human build mental representations for concepts around us in our daily life.
 humans to respond with any word associated with the given cue word.
It makes the collection process become less effortlessly for paricipants and elicits more deeply and broadly associations in human minds without the speficied relation restriction.
By crowd-sourcing, large-scale free word association resource SWOW \citep{Deyne2013BetterEO,Deyne2019TheW} is curated.
This resource contains valuable knowledge, surprsingly, its utility for natural language understading has not been explored.

In this paper, we explore the utility of commonsense knowledge collected from free associations for commonsense question answering, by representing it as knowledge graph embeddings and applying it to downstream tasks.
Also, we compare it to ConceptNet to study which one is more useful for commonsense question answering.
Our main contributions are:
\begin{itemize}
    \item We show that the free word association network SWOW alone can be used for answering commonsense questions.
    \item We show that incorporating SWOW improves on baseline question-answering models on a couple of commonsense question answering tasks.
    \item We compare the utility of ConceptNet and SWOW from both qualitative and quantitative aspects, showing that SWOW brings more improvements to xx tasks, while ConceptNet contribute more to yy tasks.

\end{itemize}

\section{Commonsense Knowledge}
\begin{table*}[t]
    \centering
    \begin{tabular}[width=\textwidth]{lccccccc}
    	\toprule
    	\textbf{KG} & \textbf{Triples} & \textbf{E} & \textbf{R} & \textbf{D} & \textbf{ID} & \textbf{EE} & \textbf{RE} \\
    	\midrule
    	ConceptNet & 2,403,453 & 799,273 & 17 & 3.76e-06 & 2.95 & 4.73 & 1.0e-4\\
        SWOW & 1,595,556 & 124,627 & 2 & 1.03e-04 & 11.11 & 1.11 & 1.79e-05\\

    	\bottomrule
    \end{tabular}
    \caption{\label{tab:statitics_cn_rel17_sw} Statistics of ConceptNet5.6 and SWOW. To densify the graph for better knowledge graph training, 47 relation types are merged into 17 types \citep{Lin2019KagNetKG}. E and R are unique entity and relation numbers. D is the graph density. ID is the average in-degree nodes. EE and RE are the entity entropy and relation entropy.}
\end{table*}

\subsection{ConceptNet Knowledge Collection}
ConceptNet \citep{Speer2017ConceptNet5A}, originated from Open Mind Common Sense (OMCS) (Singh, 2002) project launched in 1999, is presently the largest knoweldge graph containing relational commonsense knowledge.
To expand the knowledge beyond OMCS, the recent version, ConceptNet5.5 (Speer et al., 2017) integrates knowledge from several additional sources, such as Wiktionary.
Each node in the graph is an arbitrary natural language phrase or word, and each edge is labelled with a semantic relation, e.g., `UsedFor, PartOf'.
These semantic relational facts are useful for answering commonsene questions as can help determine the plausiabilty of a given answer choice.

\subsection{SWOW Knowledge Collection}
SWOW \citep{Deyne2019TheW} is currently the largest word association collection in English, collected between 2011 to 2018 from over 9,000 participants.
It captures general associations among English nouns, adjectives, verbs and adverbs.

After pre-processing, we obtain xxx unique cues and xxx responses with xxx edges.
The current released SWOW-EN consists repsonses from 100 participants per cue, and each participant can generate at most 3 responses. This process resulting in a graph that is densely connected.
As SWOW is not labelled with relation types, to train it with knowledge graph embedding techniques like TransE, we label edges with directionality of association, including \textit{forward association, backward association, and mutual association}.

\subsection{A surface Comparison}
The difference between \texttt{ConceptNet} and \texttt{SWOW} is that the former contains semantic relations, such as `PartOf, UsedFor, IsA', while the latter contains more general associations among words under the broad category of `use', `form' and `meaning' \citep{Nation2001LearningVI}.
They share common nodes and also have their own biases.

ConceptNet
    花了很多年构建,
    稀疏,automatic construction,
    包含relations (适合commonsense reasoning的),

SWOW: 目前是7年,
    比ConceptNet密 (尤其是cue words的connection 以及single word的connections[collocations比ConceptNet多]),没有relations, crowdsourcing
    increase scability

In contrast to ConceptNet,

SWOW could complement ConceptNet in terms of
In light of this, exploring whether word associations network can be used as an usable

SWOW and ConceptNet have several similarities:
(1) they both contain general commonsense knowledge ();
and
(2) concepts in the two graphs are both in the form of natural language and their ambiguities are not diffierentiated;

On the other hand, SWOW and ConceptNet differ from each other in some aspects:
(1) ConceptNet is a relational semantic network, however, SWOW doesn't contain relational edges and the latent realtion types of word association could be different from semantic relations in ConceptNet;
(2) as their processes of development differ (manually handcrafted vs. automatically generated), intuitively ConceptNet has higher quality and robustness;
(2) while ConceptNet contains more compound concepts
while WordNet focuses on formal taxonomies of words, ConceptNet focuses on a richer set of semantic relations between compound concepts [6];
and (3) WordNet differentiates ambiguous meanings of a word as synsets, however, ConceptNet bears
ambiguity of commonsense knowledge in its concepts and relations.

\section{Models}

To study the utility of of a knowledge graph to downstream commonsense reasoning tasks, we pre-train it with knowledge graph embedding techniques and use it to augment vanilla question answering models.

To examine the effectiveness of a knowledge graph under different settings, three groups of encoders are used for representing a pair of question and answer, including graph model and text+graph model.

\textbf{KG models}
To investigate whether knowledge from a knowledge graph alone can be used to answer questions, we use KG-aware models to encode question-answer relevant knowledge retrieved from a knowledge graph for question answering.
Naturally, a KG-aware model generates a knowledge embedding $\mathrm{k}$ to represent structure information for classification.
To compare ConceptNet and SWOW throughly, we employ different types of KG-aware models, ranging from static KG-aware models, i.e., GconAttn and RN, to dynamic KG-aware models i.e., PG-Global and PG-Full.

\textbf{GconAttn} \citep{Wang2019ImprovingNL} aligns grounded question concepts and answer concepts with concept-level attention and pooling strategy to generate a fixed knowledge embedding $\mathrm{k}$.
This model only incorporates graph structure and entity representation for modeling and without assuming the access to relations, thus it is a relation free model.

\textbf{RN} \citep{Santoro2017ASN} generates knowledge embedding$ \mathrm{k}$ by performing path-level attention over paths retrieved from a KG for each pair of question-answer entities.
As paths serve as structure message, RN is an relation dependent model.

\textbf{PG-Global} \citep{Wang2020ConnectingTD} generates dynamic knowledge embedding $k$ with a pre-trained path generator, which is a fine-tuned GPT-2 with  multi-hop paths sampled from a KG.

\textbf{PG-Full} \citep{Wang2020ConnectingTD}, a strong model for commonsense question answering, unifies static knowledge embedding from RN and dynamic knowledge embedding from PG-Global.

\textbf{Text Models}
We explore a range of text encoder, including a simple encoder BiGRU and pre-trained language models BERT-Base and Albert.
BiLSTMs learns the contextual representations for a pair of question and answer.
The text encoder takes input of a single sequence concatenated by a question and an answer and outputs a fixed dimension vector for answer scoring.
In addition, previous researches have shown that fine-tuning pre-trained language models like BERT, achieve promising performances on commonsense question answering tasks.
So, we also use BERT-Base and RoBERTa as sentence encoders to examine
whether knowledge from external knowledge graph like SWOW can bring information beyond pre-trained language model and textual information.

\textbf{Text+KG Models}
To augment text representation with knowledge representation, we experiment with adding graph models on top of text models. Specifically, we concatenate  representations from the text encoder and graph encoder for final classification.

\section{Experiments}
\subsection{Datasets}
\begin{table}
    \begin{adjustbox}{width=0.5\textwidth}
    \small
    \begin{tabular}[width=\textwidth]{lcccc}
        \toprule
         & Train & Dev & Test & C \\
        \midrule
        CommonsenseQA & 8,500 & 1,221 & 1,241 & 5 \\
        OpenBookQA & 4,957 & 500 & 500 & 4 \\

        \bottomrule
    \end{tabular}
    \end{adjustbox}
    \caption{\label{tab:datasets_statistics} Statistics of three commonsense question answering datasets.* uses the split from \citep{Lin2019KagNetKG}. C denotes the number of choices.}
\end{table}

We consider two commonsense question answering datasets for experiments.
They all are  multiple-choice style, and crated to prob the commonsense reasoning ability of a model. Statistics of the two datasets are shown in Table~\ref{tab:datasets_statistics}. \\
\textbf{CommonsenseQA} \citep{Talmor2019CommonsenseQAAQ} contains commonsense questions generated based on sub-graphs extracted from ConceptNet by crowd-workers.
Answering questions in this dataset is challenging as it requires various commonsense skills, and it's difficult to distinguish distractors from the correct answer because they often share same semantic relations as the key concepts in a question.

\textbf{OpenBookQA} \citep{Mihaylov2018CanAS} contains questions along with elementary level science facts built from open book exams. It requires a model to combine book knowledge and commonsense knowledge for reasoning. In this work, we ignore the provided facts and use knowledge extracted from ConceptNet and SWOW as background knowledge.

\subsection{Experimental setup}

\paragraph{KG Embeddings}
We employ pre-trained language model RoBERTa to extract entity embeddings.
Specifically, for each entity $e_i$, we feed the sequence of $['CLS', e_i, 'SEP']$ to RoBERTa and then use the representation of 'CLS' token from the last layer to represent its entity embedding.
We use relation embeddings pre-trained on a knowledge graph with TransE \footnote{TransE is trained with OpenKE https://github.com/thunlp/OpenKE}.

\begin{comment}

\subsection{Results on Commonsense Question Answering}
In the next section we will answer the following questions: \\
\textbf{Q1}. Can knowledge graph embeddings trained on SWOW be used to perform commonsense question answering?\\
\textbf{Q2}. Can knowledge graph in SWOW bring additional improvements on text encoder?\\
\textbf{Q3}. How does knowledge retrieved from ConceptNet and SWOW is different?\\

\subsubsection{Ablation}
To better understand why SWOW can achieve similar improvements on three datasets.
We study the difference of ConceptNet and SWOW in terms of the coverage and representation of mentioned concepts over three datasets.

For the coverage, we did statistics for mentioned concept types and tokens over three datasets. We found that even though SWOW has lower type coverage than ConceptNet, the token coverage is almost identical, which could explain that why they obtain similar improvements.

Furthermore, to investigate how coverage influences the performance to downstream tasks.
For both concepts in SWOW and ConceptNet,
We use the

One concern of using SWOW as a knowledge base is that its concepts coverage is much lower than ConceptNet, which may weaken its influence to downstream tasks.
So we compare the statistics of mentioned concepts from two knowledge graphs on three datasets, shown in Table~\ref{tab:retrieved_concepts_statistics}.
We can see that even less mentioned concepts are retrieved from SWOW, it still performs better than ConceptNet, suggesting that its densely connected neighbors may facilitate the higher quality concept embeddings.

\begin{table}

    \caption{\label{tab:retrieved_concepts_statistics} Statistics for average question and answer tokens of three datasets.}
\end{table}

\section{randomless}

one insight:
SWOW could be an adequent alternative to ConceptNet

ELMO
Glove
BERT
how much knowledge
what kinds of knowledge beyond text are in ConceptNet and SWOW?
ConceptNet is a big, it doesn't translate to other language easily
SWOW is simpler architecture, SWOW is easier to sc

Some and interpreable

Which of the two is best in commonsense question answering tasks?

-------------

we show that X improves upon Y (because Z)

. In this paper, we introduce the ConSeqNet
framework, which enables the use of various kinds of external knowledge bases to retrieve knowledge relevant to
a given NLI instance, by retrieving information related to
the premise and hypothesis.

We describe our novel architecture and demonstrate its use with a specific external knowledge source – ConceptNet – and evaluate its performance on
two other sources, WordNet and DBpedia.
We compare the performance of three distinct approaches to augmenting the
knowledge used to train for and to predict entailment relationships between given pairs of premises and hypotheses:
graph-only, text-only, and text-and-graph. Using both qualitative and quantitative results, we demonstrate that introducing graph-based features boosts performance on the NLI
problem, but only when text features are present as well. Our
system has a competitive performance (accuracy) of 85.2\%
on the SciTail entailment dataset, which is derived from science domain question answering datasets.

Q2. Can Knowledge Graph embeddings trained on SWOW bring additional improvements on text encoder?
Two aspects:
\begin{enumerate}
    \item Why adding GconAttn on the top of BiLSTM drops the performance?
    BiLSTM + OBQA
    BiLSTM + GconAttn + OBQA

    Text: Glove + OBQA/CSQA
    Glove+GconAttn +

    \item How can I verify this with a simple model?
\end{enumerate}

key problem:
I need a baseline for CSQA. (ESIM, BiLSTM, or glove ) and then proove add kg on the baseline can improve the performance.
the official code provides the ESIM code, (SWAG also provide the glove and number batch code)

I will use the split of (Lin 2019) on ESIM, get the baseline results.
\end{comment}


A new Sesame Street parody has emerged on the internet, this time the long running educational program has decided to give Stranger Things the Sesame Street treatment. The video, titled, "Sharing Things," has our old friend the Cookie Monster playing the part of a much less vicious, but still very hungry 'Cookiegorgon' in search for yummy Halloween treats.
The video is a lesson in sharing, as once the Cookiegorgon comes to our world from the 'snackside down', and selfishly eats all the treats, he is taught by the other kids about sharing. It is 'strange' how the right accessories and hair can make the puppets look like the characters. The video makes many references to the show and even takes some friendly jabs at Barb and in the nicest way possible, makes light of the addition of Eleven's sister, Eight.
Sesame Street has taken on other shows such as The Walking Dead, Game of Thrones, and movie franchises such as The Lord of the Rings, Twilight and Harry Potter. I enjoy these videos and think they provide a nice way for younger kids to experience the great shows that we adults do, albeit, in a toned down and more kid-friendly manner.


This is what politics has become in 2018.
This is not a joke. We just wish it was.
"I want to be your state representative, I want to be your public servant and I want to be your [N-word]," Greene says with a straight face in a video he posted on his campaign website and Facebook page.
"We need to unify as a society, and we need to stop dealing with trivial matters," Greene told the Tribune.
But how using the N-word will make the electorate train their brains to zero in on the issues instead of his usage of the derogatory term is beyond most people's ability to do.
Green, an organic farmer, wants to unseat the incumbent so that will be able to focus on judicial corruption on a variety of fronts, which includes locking up folks wrongly, applying the laws disproportionately on minorities and prosecutors who incarcerate people just to advance their own careers.
And, of course, there is the issue of unity.
Few people can perform the mental gymnastics to understand how the N-word will help unify anyone — except the extreme right-wing faction of the country.
Greene, by the way, is not endorsed by either of the major political parties nor any organization. And we can clearly see why.


The music by Tyler Imbrey & The Ghost Review is by turns harsh and sweet, downcast and uplifting, angry and joyous. They also tend to be smart, both lyrically and musically. It's kind of their trick, to insinuate themselves slowly over time, appearing at first to be more meditative and less hooky than they actually are. TIGR's recently released album, "Misgivings" is direct with both its words and sounds. That makes sense, given these precarious, and confusing times: There's not a lot of room for resignation or even patience when it feels like the world is falling apart. Can those things be spun into relevant, redemptive music? That's tricky business, but in the case of Tyler Imbrey & The Ghost Review, turmoil is a potent fuel, and "Misgivings" feels essential, connected, and unflinching.
"Misgivings" feels musically rich and textured. Alongside some late-night bar-room piano ('Rolling Over Me') and seriously moreish guitar riffs ('Lost My Head') is a constant swirl, pulse, shimmer of assorted acoustic resonance and swooping electric tones puncturing and breaking through the density of the sound.
This is unquestionably one of those albums that merits headphone listening, with so many brilliant little details – musical doodles and squiggles – that emerge over time and with the kind of close listening that it demands.
Along with the darker edginess that runs through the album, there's also a tangible feeling of uplift, sometimes of sheer euphoria in this music. Those melodic and soaring moments where Tyler Imbrey's vocals just lift a few tones are one of the project's greatest pay-offs. And they are most impressive on tracks such as "Poison", "The Town", "Old Car" and "Alarm".
But all the things that make this band a real treasure can be found here – the big, bold beauty of the melodies, the detailed perfection of the music, and the vivid storytelling. There's really so much here worth hanging on for.
There are precious cathartic moments here, as on "Dear John". The track begins with a skittering rock beat and a cooing melodic vocal. It soars out of the mire into a hugely infectious chorus, with Tyler Imbrey driving it in emphatic fashion.
His lyrics cut to the bone before the song is whipped up to a heady brew of stunning drumming and cacophonous orchestration and then back again. Masters of profound introspection, flecked with the anthemic thrust, Tyler Imbrey & The Ghost Review do a blend of indie-rock and alternative-folk better than many others.
What strikes a chord is the band's proficiency in butterflying laments into affecting existential meditations. There is also an understated urgency in the plurality and density of the arrangements. And while Tyler Imbrey has accommodated a short story collection of voices, messages and characters, there seems to be ghosts of himself veiled in each. The album reads as a wry take on the malleability of worldview and personal perception.
"Misgivings" might not make Tyler Imbrey & The Ghost Review massive mainstream chart toppers overnight, but I doubt that's what they're after anyway. The steady, organic growl of a band who get under your skin and take hold is very much in vogue here. They're surely in this for the long run.
Matt Rouch: "The Beautiful and the Damned" grab your heart and squeezes!


Throughout the site and this document, the terms "we", "us" and "our" refer to James McAllister Online. James McAllister Online offers this website, including all information, products, tools and services available from this site to you, the user, conditioned upon your acceptance of all terms, conditions, policies and notices stated here.
In no case shall James McAllister Online, our directors, officers, employees, affiliates, agents, contractors, interns, suppliers, service providers or licensors be liable for any injury, loss, claim, or any direct, indirect, incidental, punitive, special, or consequential damages of any kind, including, without limitation lost profits, lost revenue, lost savings, loss of data, replacement costs, or any similar damages, whether based in contract, tort (including negligence), strict liability or otherwise, arising from your use of any of the service or any products procured using the service, or for any other claim related in any way to your use of the service or any product, including, but not limited to, any errors or omissions in any content, or any loss or damage of any kind incurred as a result of the use of the service or any content (or product) posted, transmitted, or otherwise made available via the service, even if advised of their possibility. Because some states or jurisdictions do not allow the exclusion or the limitation of liability for consequential or incidental damages, in such states or jurisdictions, our liability shall be limited to the maximum extent permitted by law.
Our policy lasts 30 days unless otherwise stated. If 30 days have gone by since your purchase, unfortunately we can't offer you a refund or exchange.
To be eligible for a refund, you may not have downloaded or otherwise consumed the item(s) purchased. Downloaded and/or consumed items may not be exempt from being refunded at our own discretion.
To complete your return, we require you to notify us with all relevant transaction information as well as a reason for requesting a refund. Refunds may be accepted or declined for any reason at our own discretion.
If you purchase one of our products from a third-party service, we can not refund your purchase. The responsibility of handling returns or refunds is provided by the service in which your purchase is made.
Sales items will be refunded only up to the amount in which you paid. You will not be refunded the full retail price of an item if the item was purchased during a sale, or with the use of a coupon code.
You agree to indemnify, defend and hold harmless James McAllister Online and our parent, subsidiaries, affiliates, partners, officers, directors, agents, contractors, licensors, service providers, subcontractors, suppliers, interns and employees, harmless from any claim or demand, including reasonable attorneys' fees, made by any third-party due to or arising out of your breach of these Terms of Service or the documents they incorporate by reference, or your violation of any law or the rights of a third-party.
These Terms of Service are effective unless and until terminated by either you or us. You may terminate these Terms of Service at any time by notifying us that you no longer wish to use our Services, or when you cease using our site or our products.
The laws of the United States and the State of Washington will govern this Agreement, without reference to rules governing choice of laws. Any action relating to this Agreement must be brought in the federal or state courts located in Whatcom County, Bellingham, Washington and You irrevocably consent to the jurisdiction of such courts. You may not assign this Agreement, by operation of law or otherwise, without Our prior written consent. Subject to that restriction, this Agreement will be binding on, inure to the benefit of, and be enforceable against the parties and their respective successors and assigns. Our failure to enforce Your strict performance of any provision of this Agreement will not constitute a waiver of Our right to subsequently enforce such a provision or any other provision of this Agreement.


text:
http://tvtropes.org/pmwiki/pmwiki.php/Series/NoGoodNick
Series / No Good Nick
Create New - Create New - Analysis Characters FanficRecs FanWorks Fridge Haiku Headscratchers ImageLinks Laconic PlayingWith Quotes Recap ReferencedBy Synopsis Timeline WMG
No Good Nick is a Netflix original Dramedy Sitcom centering on Nicole "Nick" Franzelli, a teenage Con Artist who claimed to be a Long-Lost Relative whose parents recently died to insert herself into an upper-class family, The Thompsons. The mother Liz (Melissa Joan Hart) runs an Italian restaurant and father Ed (Sean Astin) works at a bank, while their children Jeremy and Molly are high school aged. Nick's real goals are to use their connections to scam them out of money. However, she finds herself growing increasingly attached to them as she spends more time in their home.
Part 1 premiered on April 15, 2019. Part 2 premiered on August 5, 2019.
WARNING: Despite being a sitcom, No Good Nick has a heavily serialized plot with many dramatic reveals and twists. Here there be spoilers.
This series includes examples of:
Actor Allusion: The school has a 90's-day theme which Ed and Liz attend, giving an opportunity for Melissa Joan Hart to dress up like Clarissa again.
Ed builds tree sculptures out of Lord Of The Rings-books.
Jeremy calls Molly "Little Hobbit."
And This Is for...: Molly ends up estranged from her Volunteer Squad friends due to some unresolved issues and a private rant complaining about them that was made public. As she falls into a funk, Ed sets up a station in the garage where she can smash old appliances and release pent up emotions. Once she gets going she starts listing off all the bad stuff that had been happening to her and the things they did, then she blurts out how angry she is at herself for being a crappy friend and collapses in tears.
Anti-Hero / Anti-Villain: The whole main cast. They all did/do underhanded, unethical things for their family, and while the show portrays them as fundamentally good people whose love is genuine and positive, it doesn't let them off the hook and makes them confront the consequences of their actions and how they affect other people.
Batman Gambit: Most of Nick's cons are pretty simple tricks, but the finale of part two takes the top of the list. Ed was arrested for the theft of the necklace, and she comes clean to the family promising she had a plan to make it right, but they leave her to the police. By telling Sam and Dorothy the necklace was still at the Thompsons, they co-op her for one last infiltration before ransacking the entire home. Nick silently reveals her plan to the Thompsons and convinces them to play along with a spiked drink. With Sam and Dorothy inside the house, everyone wakes up and the police arrive, as she cut a deal with them offscreen. In one swoop she got Sam and Dorothy to take the brunt of the necklace theft, unchaining her from them, getting Ed completely off the hook while also getting her dad immunity. It also served to restore her relationship with the Thompsons.
Becoming the Mask: Nick's scheme against the Thompsons was entirely motivated by revenge, but being part of a loving family who think they took in an orphan, making them act extra kind to her, she can't help but appreciate having a sister and brother with parents who are Happily Married.
Beneath Notice: Being a 14 year old girl, almost no one suspects that Nick is the one pulling off various scams and heists (and in most cases people don't realize they have been scammed). Those that do are dismissed as paranoid.
Beware the Nice Ones: Of the entire cast Ed is the most genuine and friendly, evoking a harmless goofball. But a few occasions show him as not being a slouch in a fight "You just got Ed-ucated." Also, when Nick's story is exposed and the family deals with the truth, he turns out to be the most unforgiving and held on to his anger longer than the others.
Birds of a Feather:
Nick and Will quickly bond over having parents in prison and the trauma that it conveys. Later, they also bond over being used by Sam and Dorothy to scam someone they've come to really care for
Jeremy and Eric have a lot of the same interests that develop into mutual attraction.
Bookends:
The first episode shows Nick as trying to drug the family so Sam and Dorothy could ransack their home, but pulls back claiming to be doing a larger con. The end of part two Nick pretends to drug them and they pretend to be knocked unconscious so that Sam and Dorothy will enter the house. Once inside, the family wakes up and the police arrive.
In the last episode of part two, when Nick's story is exposed, Ed quickly tosses her to the police saying "This girl stole the necklace!" Once she resolved all the problems she caused, she packs up and leaves while leaving a letter behind explaining everything including her guilt. When the family reads the letter, Ed is the one to say "Let's get our girl back," letting go of his anger.
Both Sides Have a Point: In "The Fool's Errand", while Tony has every right to be angry at the Thompsons for doing unethical things that ruined his restaurant, Nick points that out it was his own fault that he ended up in debt to the mob for borrowing from them, and in prison for robbing a convenience store.
Bumbling Dad: Downplayed with Ed, as he's seen as the more laid back, goofier parent next to Liz. He's actually quite level headed and empathetic to others, although his hands-off style has caused some problems.
Calling the Old Man Out: In "The Fool's Errand", Nick calls out Tony for dragging her into his revenge scheme against the Thompsons, pointing out that, regardless of what they did to him, the decisions that landed him in prison were his and his alone.
Chekhov's Gun: Plot points, conversations and dilemmas will be brought up only to be circumvented in some way, but multiple episodes later will emerge as the lynch pin of another ploy. In the poker night episode it's mentioned there are security cameras around to keep track of things, but much later Nick uses the recording of Molly complaining about her friends to utterly decimate her relationship with them.
The Cobbler's Children Have No Shoes: Liz is so engaged with her restaurant she rarely has dinner with her family, let alone cook for them. It was shown to be even worse in the past, but she started making meals again when Nick came along.
Consummate Liar: Nick is especially good at deceiving people on the fly, which is why her foster parents went along with the scheme. Though this doesn't fly when a social worker visits, who can tell Nick is putting up a front and acting nice to stop the questions. In turn, Nick manages to win her over by telling the truth, albeit leaving out important details.
Continuity Nod: All over the place. The show has a surprising attention to continuity, with what seems to be one-off characters and jokes being referenced or showing up again later.
Nick mentions that a girl in her old soccer team always pretended that she was playing Quiditch. Later, during a flashback, Riley mentions that same girl while warning Nick that she's wearing her Hufflepuff robes.
The Reveal of Nick's plan in the poker episode relies on the viewer remembering Todd from previous episodes.
In the flashback to Franzelli's, we see at the restaruant several people who we've seen (or will see) help Nick in her schemes, justifying how she knows them and gets favors from them.
Conveniently an Orphan: Exploited by Nick, the Thompsons are too sympathetic to her story to question it too deeply.
Couch Gag: The intro consists of 4 pictures: The Thompsons pre-Nick, The Thompsons with Nick, Nick happy by herself, and finally Nick sad by herself holding an item related to the following episode.
Deconstruction: Of the "family first" Aesop, done in as brutal and ruthless a way as can be done in a children's show: The characters do whatever it takes for their families, which is often underhanded and results in the suffering of innocents. It's most notorious when mobsters, arguably the only characters without any redeeming qualities, justify their actions by saying they just have to do it for the (crime) family.
Didn't Think This Through: Nick is very smart and can easily manipulate circumstances to get what she wants in the immediate future, but her schemes almost invariably have unintended consequences that complicate the situation, because she fails to realize that events don't occur in a vacuum and her "solution" affects more than just the current moment.
She manages to make a big payday on a "Go Fund Me" fake charity scam. Too bad she didn't realize that, as a minor, she can't open an account without an adult co-signer so she can never actually access the money.
Dramedy: Despite being a sitcom with a Laugh Track, the show frequently depicts criminal actions with appropriate seriousness.
Dysfunctional Family: Downplayed with the Thompsons, every member has fairly realistic conflicts stemming from poor communication, conflict of interests and long-stemming resentment. There are arguments that spreads across multiple episodes, but every so often they have a fun family activity that shows them being happy together.
Easily Forgiven: Not exactly easily, but after learning Nick's true story the family was quick to leave her to the police. But learning their culpability in what happened to the Franzelli's left them with complicated feelings. Nick pulls off one last trick to make things right, and also set things in motion to fix the damage she caused. Although Nick tries leaving town, they follow her to take her back.
Family of Choice: Pretty much literally. Nick had to convince them she was a Long-Lost Relative, but she came to view Liz and Ed as her parents and they came to see her as their daughter.
"Fawlty Towers" Plot: Any given episode has Nick trying to push through a scheme, all while trying to keep the original lie going on.
Foreshadowing:
Eric is very protective of Jeremy and has trouble remaining impartial during the poker tournament we later see them kissing.
During the escape room Eric "corrects" the mention of the moon landing for "alleged moon landing", several episodes later we learn that he's a conspiracy nut.
Almost all the elements revealed in the Origins Episode were established piecemeal across the previous episodes, including Nick knowing a lot about Italian food, her dad owning a restaurant and him owing money to the mob.
After the Wham Shot at the end of "The Pigeon Drop" reveals that the guy Nick gave money to isn't really a defense attorney, the Cold Open of the next episode has her father tell her that the lawyer took his case. This is an early indicator that her father has been lying to her about what the money is for.
Nick offers Molly pizza and Molly tells her she is gonna leave a negative Yelp review without even tasting it which Nick looks shocked at. Molly left bad Yelp reviews for Tony's restaurant in the past.
Free-Range Children: Few, if any, of Nicks plans would work if there was any real accountability to where she was at any time of the day or night. She regularly visits contacts, criminals, her foster parents and even her dad in prison without the Thompsons suspecting anything. Jeremy and Molly are similar, with Jeremy trying to get his own car a plot point, but are certainly not as busy as Nick.
Gambit Roulette: Nick initially started her scheme with the help of her actual foster parents, but she is actually manipulating them for a revenge plan against the Thompsons developed by her dad, who is in prison. Trying to juggle all the lies takes up a lot of her time.
Gayngst: Averted, Jeremy doesn't seem to have much of a problem with his identity, he just wants to come to terms with it by himself before opening up, and when he finally does nobody seems to care except to thank him for the trust.
Good Feels Good: As much as Nick feels compelled to continue her schemes for revenge, she doesn't take much joy in seeing any pain she causes. Conversely, trying to (secretly) make up for what she did has a much stronger net positive, though it takes her time to figure out what she wants.
Graying Morality: The show initially seems to have a very clear morality, with the Thompsons being a kind and loving family and Nick a villain trying to steal from them. We soon learn that Nick is being pressured into it by her father, and that she genuinely cares about the Thompsons. However, the morality gets maximum grey in The Italian Job, where we learn that The Thompsons are far from innocent, and the Franzellis are completely justified in blaming them for the unfair ruin of their livelihood, even if they're still wrong in wanting revenge.
Happily Married: Whatever conflict Liz and Ed may have, it's clear that they love each other very much and they work through all their problems as a team.
Heel–Face Turn: Nick starts out with the intention of robbing the Thompsons blind, but warms up to them and reconsiders her scheme over the course of the series, ultimately abandoning it at the end of Part 2.
Heel Realization: The show eventually reveals itself to be on Grey-and-Gray Morality, as everyone does or had done some morally sketchy things that had a negative impact on other people. Over the course of the series, though, the only difference between the good and bad characters is that the good ones eventually realize that they're the villains in someone else's story, and show genuine regret and growth from this realization.
Most notably, when They learn how their actions led to Nick's dad landing in jail, the Thompsons are struck by how they all ended up becoming so greedy and obsessed with success that they ruined a man's life rather than accept a little friendly competition.
Hidden Depths:
For a small time, barely competent crook with a BA in sports administration, Todd is a surprisingly good therapist.
Inverted, then subverted with Eric. He and Jeremy bond over similar interests, but Jeremy is then so horrified by Eric being a conspiracy nut that he dismisses him outright. Later Jeremy comes to realize that being a conspiracy nut doesn't change the fact that Eric is a kind, fun guy, and it's implied that they might give it another chance.
Jim is a big burly guy who functions as a sort of assistant/bodyguard for Jeremy, he initially seems to be Dumb Muscle, but through the series he's shown to be a skilled spy, a sensitive reader, a perceptive listener and ultimately a Consummate Professional.
High-School Hustler: Nick in a nutshell, woven in as literally as possible.
Hypercompetent Sidekick: Jeremy's friend Jim is a level headed sidekick trusted with running odd jobs and gathering information. Notably he is the one character who always comes through, and in one episode Nick bails on a goal knowing that Jim won't overlook what is going on.
Identical Grandson: Nick comes across a picture that was initially mistaken for being her, but was actually her mother as a teenager with Sam and Dorothy.
Idiosyncratic Episode Naming: Every episode is named after a real world scam type, and typically demonstrated in the episode itself. The titles also begin with the article "The".
In Love with the Mark: A non-romantic example in Nick who develops a sincere attachment and appreciation for the Thompsons. Played completely straight with Will
Instant Expert: Subverted. When Nick is gathering votes for Jeremy we see her telling Blatant Lies about him to several clubs. She concludes her talk to the soccer team by keeping the ball in the air in a day display of skill that's very impressive for a girl with no real interest in the sport. However, later in the season we learn that she used to be in the soccer team of her old school. Which makes her skills justified.
It's All About Me: This is Molly's primary trait, dedicated to social causes but especially how it raises her social media presence. When the show starts breaking down its characters it turns out almost every one of the Thompsons has this problem, but in the sense that they were so focused on their own family they were blinded to what they were doing.
Jerk with a Heart of Gold: Downplayed with Sam and Dorothy, Nick's actual foster parents. They're shown to genuinely care for their foster children, but the show doesn't shy away from the fact that turning children in your care into tools for your scams is very wrong and doesn't portray them as good people. If anything it exaggerates their hypocrisy.
Locked Out of the Loop: It turns out every member of the Thompsons was totally unaware of how the others were acting against Franzelli's, believing they alone were doing just one minor thing and the rest of the events were pure accidents. Each is rocked to hear what the others did with Liz protesting she had no idea of any of this.
Manipulative Bastard: Nick proves to be extremely good at anticipating others behavior in order to coordinate her schemes, all while keeping herself under the radar. But the second half of part two she starts targeting the Thompsons directly, and rather ruthlessly dismantles their lives in the process.
My God, What Have I Done?: The reaction of the Thompsons when Nick lays it out how each one of them had a part in absolutely ruining her family's life. For the first time, it hits them all how what they thought were just attempts to help out Liz led to a once-good man turning to crime and ending up in prison and his daughter embarking on a journey of revenge.
Nick feels this way as well once she exacts her revenge against each of the Thompsons.
Never My Fault:
Molly. When Nick points out that she's going through the same she put other people through she dismisses the lesson and claims that it's different because this time she is the victim. The more Nick tries to force an epiphany on her the more Molly leans onto her role of "innocent victim". It's not until her run in the "Feel your feels" station that she finally breaks down and admits that a lot of her problems are just the consequences of her own wrongdoing.
Tony. While he's right about the Thompsons unfairly and underhandedly running his restaurant out of business, the fact that he's in deep debt with the mob and in prison is a result of his own bad choices.
One-Hour Work Week: Liz and Ed have jobs that in real life require a lot of time and effort, yet they're very often just hanging around the house in the middle of the day.
Origins Episode: "The Italian Job" shows how Tony ended up in prison, why Nick ended up in foster care, and why she targeted the Thompson family.
Pay Evil unto Evil: Deconstructed. Nick justifies manipulating and eventually ruining the Thompsons lives out of a belief that they conspired against her family. But actually becoming a part of their family made her realize they were just flawed people living their lives, and ultimately don't deserve what happens to them anymore than what happened to her.
Percussive Therapy: The "Feel your feels" station made by Ed (patent pending) to help Molly let out her frustration by destroying old stuff with a bat, Liz also benefits from it.
Pitbull Dates Puppy: Liz is the stronger personality in the family, while Ed is laid back and often in his own world.
Political Correctness Gone Mad: Exploited by Nick in the costume dance. She pressures Jeremy into letting the volunteer squad fine people with insensitive or offensive costumes, then she finds a way to interpret every costume as insensitive or offensive, such as fining a ninja for cultural appropriation, or a vampire for being insensitive to haemophiliacs. Though, even she draws the line at Jeremy being excluded from his own dance for dressing as Thomas Jefferson (albeit for personal reasons, as she doesn't want him to have further reason to distrust her).
Precision F-Strike: Despite the subject matter the show as a whole is not too dissimilar in the type of dialogue and conversations you would see in a Disney Channel sitcom. But one episode Nick comes across Todd practicing to play a particular role and asks "What the hell was that?"
Properly Paranoid: Jeremy is very suspicious of Nick and suspects ulterior motives in everything she does. Of course, Nick is lying about everything and has ulterior motives for an her actions.
Pun-Based Title: "Nogoodnik" is an archaic term for someone untrustworthy and prone to criminal behavior, much like Nick, pre-Character Development.
Protagonist-Centered Morality: Defied. One of the main themes of the Story Arc is that people outside of yourself and yours also matter and whatever bad thing you do to them, even if it is to help your loved ones, is wrong and has consequences. Nick's actions are treated as maybe clever, but never okay, and it's revealed that the Thompsons have done some shady things without realizing how immoral it was.
Recurring Character: There is a sizable group of recurring characters between school, friends and local criminals. Notably, Todd randomly showing up at the poker raffle serves as The Reveal to Nick's real plan.
Revenge: Nick's true motivation for targeting the Thompsons. Her father, Tony, was once the owner of a successful Italian restaurant, but when Liz Thompson started her new restaurant, she sought to eliminate the competition by poaching their costumers. Each member of the family had a hand in the subsequent downfall of Franzelli's: Liz redesigned her menu to make the same kind of food as Franzelli's, but at a lower cost, Molly and her friends posted fake bad reviews on Yelp, Jeremy interfered with a campaign to bring back customers by stealing menus that Nick and Tony distributed, and Ed used his position as a bank manager to deny Tony an extension on a loan. Driven to desperation, Tony took a loan from a Mafia boss, then robbed a store in an attempt to pay it back, which resulted in his arrest and imprisonment.
Rich Bitch: Ms. Chang. She's the top client in Ed's bank and uses her position to belittle and disrespect him any chance she gets.
Liz and the Thompsons in general can be like this, but to a smaller degree. Liz fought hard to build up her restaurant, proudly declaring it gourmet Italian and superior to the slop houses in town. They have their own Rags to Riches story to justify it, but it contrasts the Franzelli's restaurant, a more home cooked spaghetti and meatballs place that the Thompsons systematically sabotaged using money and social media connections. This forms the basis of Nick's revenge scheme against them.
Running Gag:
Ed using the word "literally" and another character promptly telling him this isn't how to use that word.
Molly or her friends saying "That's my thing!"
The Rival:
For Molly it's Becky, who also wants to be the leader of the volunteer squad
For Jeremy is Lisa, the former vice president of the student council who also wants to be the new president.
Sequel Hook: The final episode of season 2 does a good job of tying up loose ends, with the main characters facing their flaws and the consequences of their actions and starting on their way of making amends. However, in the last few minutes we see a picture of Nick's mom with Sam and Dorothy, setting up a mystery for a possible next season.
Skewed Priorities: The volunteer squad, and particularly Molly and Becky, seem more interested in exploiting causes for their own self promotion than into making a real difference in the world. It's usually played for laughs, but in later episodes it's shown with more seriousness as a character flaw.
Something Only They Would Say: Nick figures out that Will is a spy for Sam and Dorothy because he makes a remark about "the juice" not being "worth the squeeze", a phrase they particularly like to use.
Soapbox Sadie: Molly is pretty vocal about her passion for social and environment causes, being part of the "Volunteer Squad" at school, though little effort is put in to making it seem like she isn't in it for the social media attention she gets. Many of Nick's scams involve utilizing their connections or skimming off their fundraising.
Spy Speak: Repeatedly played for laughs, either being unnecessary or one of the parties just not getting the hint.
Surprisingly Realistic Outcome: A number of Nick's schemes fall apart due to not knowing the exact details needed to make it pay off the way she expects them to.
She was able to steal a bunch of expensive wine from Liz's restaurant by switching the labels with cheap wine. Unfortunately, the value of wine is almost entirely in the label itself, so she just ended up with a bunch of blank wine bottles with functionally no commercial value.
She figured out the passwords to get administrator access at the bank, but learns that even the bank president can't make large transfers without secondary confirmation or being noticed. She would never be able to move funds and withdraw it fast enough.
Setting up a false "Go Fund Me" page she was able to acquire a large sum of crowd sourced pity money, but being a minor was unable to open a personal bank account without an in-person adult co-sign. This left the account less private than she needed it to be.
Her main deception against the Thompsons required them to believe she was a Long-Lost Relative who needed a home. Because of this set-up for the most part they ended up showering her with love and sympathy. To continue this kind of deception while dismantling their lives required a level of callous cruelty that Nick just didn't have. She had functionally set up her own Stockholm Syndrome and was Becoming the Mask.
Token Houseguest: The show's premise has a Con Artist shows up to an upper-class family's house, claiming to be a Long-Lost Relative. They let her stay with them, and she subsequently plots to scam them out of money. The show is a Deconstruction of sitcoms.
Transparent Closet: Eric was asked about how he came out to his parents. His response was that everyone already knew.
The Unfavorite: Both Jeremy and Molly at different points feel upstaged by Nick, as Liz and Ed tend to be more lenient on her and Nick, usually due to running a scam, shows a lot more enthusiasm with their interests.
Unsympathetic Comedy Protagonist: Nick takes this almost to the extreme, as the premise is she's a teenage criminal who manipulated a seemingly wholesome family into thinking she was a Long-Lost Relative and recently orphaned, solely to rob them blind. The first episode she comes close to drugging them before ransacking their home... Hilarity Ensues? But Nick pulls back when needed, trying to minimize the harm she does while also showing she is being manipulated by both her foster parents and a dad in prison.
Vengeance Feels Empty: Nick comes to realize this over the course of the series. Her vengeful scheming against the Thompsons didn't make her feel better or bring back the happy life she used to have. In "The Fool's Errand", she sets aside her hatred towards them for good.
Villain Has a Point: Invoked in-universe when the Thompsons finally discover the truth about Nick. Liz, Ed and Molly are all very upset with Nick lying to them but Jeremy reminds them how, while her actions were extreme, Nick is completely correct in how the Thompsons are responsible for ruining her father and her anger against them is justified.
Wham Episode:
"The Jam Auction" reveals the truth behind Nick needing to get money for the lawyer, and makes it clear that he is not coming home soon and his life is in danger.
"The Italian Job" revealed why Nick wanted revenge against the Thompsons and how her father ended up in jail, which put a different spin on the Thompsons and Nick's situation.
Wham Line: "We do love her" said by Ed as he reveals that he was only pretending to be knocked out.
Wham Shot: Nick goes to see her dad in prison, and he was badly beaten up by members of the mob.
Where the Hell Is Springfield?: The setting for the show is left vague, but bits of dialogue and signage implies it's in a fictional Portland, Oregon suburb called Overton.
Whole Episode Flashback: Except for the opening and closing, "The Italian Job" focuses solely on events that occurred before the start of the series.
Women Are Wiser: Subverted with Liz. She thinks she is more attentive to the family than Ed is, but her constant nagging and big personality often breeds resentment. Ed is more laid back but also more emotionally attentive, recognizing the "chore auction" as pitting their children against each other rather than incentivizing allowance.

redpajama_set_name: RedPajamaCommonCrawl


text:
Q: local mail setup to debug root cron jobs on debian I'm running the following setup:
beaglebone black [wireless version - 4.9.82-ti-r102 #1 SMP PREEMPT] -- running debian 9.3 stretch.
I've attempted to install the latest postfix mail server -- just so
I can read the debug messages from cron.
I notice that the mailque is full of emails to
root@beaglebone.localdomain.
Yet, if I try to:
> sudo mail

I get :
no mail for root

Question:
What should my root email address be ?
Obviously not root@beaglebone.localdomain!
Is there something wrong with either my postfix conf or the contab?
My /etc/postfix/main.cf looks like this:
# Debian specific:  Specifying a file name will cause the first
# line of that file to be used as the name.  The Debian default
# is /etc/mailname.
myorigin = localhost
mydestination = $myhostname, localhost.$mydomain, localhost
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
relayhost =

The root contrab looks like this:
MAILTO="root@beaglebone.localdomain"
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
#
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h  dom mon dow   command
5,35 * * * * cd /home/debian/bbbwifilogger/ && Rscript /home/debian/bbbwifilogger/graph.R &>/home/debian/bbbwifilogger/cron.log
6,36 * * * * cd /home/debian/bbbwifilogger/ && cp /home/debian/bbbwifilogger/moist.png /var/www/html/
6,36 * * * * cd /home/debian/bbbwifilogger/ && cp /home/debian/bbbwifilogger/tdegc.png /var/www/html/
7,37 * * * * cd /home/debian/bbbwifilogger/ && cp /home/debian/bbbwifilogger/dnlddata.zip /var/www/html/
3,33 * * * * /etc/init.d/networking restart
10,40 * * * * /usr/sbin/rtcwake -m mem -u -t $(date +%s -d "20 minutes") >> /home/debian/bbbwifilogger/bbblogger.log 2>&1

Thanks in advance for your advice.

A: To avoid mailbox full issue just increase email box size or rotate the mailbox.
/etc/postfix/main.cf:

message_size_limit = 31457280

redpajama_set_name: RedPajamaStackExchange