text:
Article # 5RIB6
October 2010 // Volume 48 // Number 5 // Research In Brief // 5RIB6
Precision Farming Information Sources Used by Cotton Farmers and Implications for Extension
Cotton farmers in Alabama, Arkansas, Florida, Georgia, Louisiana, Mississippi, Missouri, North and South Carolina, Tennessee, and Virginia were asked where they obtained information about precision farming. Farmers use Extension as a source of precision farming information, but in combination with other sources, including media, crop consultants, trade shows, and other farmers. Farmers using Extension as a source of information tend to be younger, with more education and higher incomes, compared to producers not seeking information from Extension. Understanding the profiles of producers using Extension resources for precision farming information may help Extension design programs to improve information delivery to clients.
Keywords: information source use, precision agriculture technologies, Extension
Margarita Velandia
mvelandia@utk.edu
Dayton M. Lambert
dmlambert@utk.edu
ajenki21@utk.edu
Roland K. Roberts
rrobert3@utk.edu
James A. Larson
jlarson2@utk.edu
Burton C. English
benglish@utk.edu
Steven W. Martin
Delta Research & Extension Center
Stoneville, Mississippi
smartin@ext.msstate.edu
Farmers using precision agriculture technologies may decrease variable costs, increase profits, and moderate environmental risks by not applying more inputs than needed (Bullock, Lowenberg-DeBoer, & Swinton, 2002; Bongiovanni & Lowenberg-DeBoer, 2004; Roberts, English, & Larson, 2002; Watson, Segarra, Lascano, Bronson, & Schubert, 2005; Torbett, Roberts, Larson, & English, 2007). Given the potential economic and environmental benefits from some precision farming practices, effective dissemination of precision agriculture information by Extension appears justified. Extension plays an important role in the diffusion of information to help individuals make informed decisions that influence profitability (Hall, Dunkelberger, Ferreira, Prevatt, & Martin, 2003). Therefore, Extension is positioned to provide information to farmers about the costs, benefits, and use of precision agriculture technologies and about how these practices can be integrated into whole farm management plans.
Demand for information about technologies to manage agricultural production systems increased with the advent of yield monitors and global positioning systems and improvements in computing power and data management (Schnitkey, Batte, Jones, & Botomogno, 1992; Swinton & Lowenberg-DeBoer, 1998; Griffin, Lowenberg-DeBoer, Lambert, Peone, Payne, & Daberkow, 2004). However, with the decline in Extension resources over recent years (Aguilar & Thornsbury, 2005; Diem, 2002; Smith & Swisher, 1986), other information providers such as crop consultants, farm dealers, trade shows, media, and other farmers are important complements for Extension and its ability to meet farmer demand for precision farming information (Just, Wolf, Wu, & Zilberman, 2002; Schnitkey et al., 1992). Farmers may choose to use a single information source, but, more likely, producers will combine various sources of information to make farm business decisions.
The study reported here examined which sources of information farmers use to obtain precision farming information and evaluated how they make decisions about the use of each source of information. We focused on complementary use of Extension and other information sources, comparing the profiles of farmers using Extension as a single source of precision agriculture information, or in combination with other information sources, including private consultants or farm dealerships, electronic media, or other farmers. Understanding the farm business and operator characteristics of precision farming information consumers and the sources of information they use to learn about precision agriculture can provide Extension with better knowledge about clientele demand for precision agriculture information. Such knowledge may motivate innovative approaches to effectively coordinate, package and deliver precision agriculture information through a variety of channels.
A survey was mailed in January, 2005, to 12,243 cotton farmers in Alabama, Arkansas, Florida, Georgia, Louisiana, Mississippi, Missouri, North and South Carolina, Tennessee, and Virginia. Of the questionnaires mailed, 200 were returned undeliverable or by farmers no longer producing cotton. In total, 1,214 surveys were suitable for analysis (10% response rate).
The survey captured general demographic characteristics, including year born, highest level of education completed, number of years spent in farming, percentage of income derived from agriculture, number of acres farmed (owned versus rented), and percentage of taxable household income derived from farming. Additionally, the questionnaire asked farmers about the precision farming technologies used on their farms and the resources used to obtain precision farming information. Additional details of the survey are found in Roberts et al. (2002) and Walton, Lambert, Roberts, Larson, English, Larkin, Martin, Marra, Paxton, and Reeves (2008).
The information sources were divided into four groups: private (crop consultants, farm dealers, trade shows), Extension, other farmers, and media. Frequencies of information sources used by respondents were enumerated based on whether the respondent reported using Extension, other farmers, private consultants or farm dealerships, media, or combinations of these sources for information about precision agriculture. Of particular interest were (1) producers who used Extension alone or in combination with other information sources (Extension Users) and (2) respondents who did not use Extension, but used one or a combination of other information sources (Non-Users). For both groups, there were eight possible combinations of information sources (Table 1). The farm business and operator characteristics of Extension Users and Non-Users were compared using independent samples t-test.
About 65% of producers used Extension either solely or in combination with other information sources. Only 2.6% of the total farmers surveyed used Extension as the only source of precision farming information. About 75% of the Extension users combined information from Extension with media sources, crop consultants, farm dealers, and other farmers.
Survey Question: Where Did You Get Your Precision Farming Information?
Information Source Combinationsa Number of Farmers Percentage
(1) Used only Extension sources 31 2.55
(2) Used private and Extension sources 37 3.05
(3) Used Extension and other farmer sources 17 1.40
(4) Used Extension and media sources 17 1.40
(5) Used private, Extension, and other farmer sources 57 4.70
(6) Used private, Extension, and media sources 25 2.06
(7) Used Extension, other farmer, and media sources 18 1.48
(8) Used Extension, other farmer, media and private sources 596 49.09
Total Extension users 798 65.73
(9) Used none of the sources considered in the survey 203 16.72
(10) Used only private sources 43 3.54
(11) Used only other farmer sources 29 2.39
(12) Used only media sources 48 3.95
(13) Used private and other farmer sources 21 1.73
(14) Used private and media sources 33 2.72
(15) Used other farmer and media sources 15 1.24
(16) Used private, other farmer, and media sources 24 1.98
Total non-Extension users 416 34.27
Table 2 compares the farm business and operator characteristics of Extension and non-Extension users. Respondents using Extension, individually or in combination with other sources, are typically younger than farmers not using Extension. Producers who use Extension tend to have more years of education than those not using Extension. A larger percentage of farmers using Extension earn more than $150,000 than those who do not use Extension (35.82% and 30.45%, respectively).
The average farm size varies between the two groups, with Extension users operating larger farms than non-users. Additionally, Extension users rent more land to farm than non-users, as evidenced by the own-to-total operated acres ratio (31.80% and 35.43%, respectively).
Variable means for Extension Users and Non-Extension Users
Variable Extension Users Non-Extension Users
Average age of producers as of 2004 (in years)*** 48.57 53.28
Proportion of farmers with a high school degree (%) 80 79
Proportion of farmers with an associate degree (%) *** 19 11
Proportion of farmers with a bachelor's degree (%) *** 37 29
Proportion of farmers with a graduate degree (%) ** 9 5
Proportion of farmers with income higher than $150,000 (%)* 36 30
Average percent of income from farming (%) 73 71
Average farm size ( in acres)*** 1469.10 1151.67
Land tenure (owned acres divided by owned acres plus rented acres %) * 32 35
*, **, and *** represent statistical significance at 10%, 5%, and 1% levels, respectively.
Producers learning about precision farming tend to use multiple sources of information to increase their knowledge about precision agriculture. Extension is one of the main sources of information, but it appears that farmers combine Extension with other information sources. About 66% of producers used Extension, and about 75% of these users combined Extension with all the other information sources, including private consultants and farm dealerships, media, and other farmers.
Users of Extension tend to be younger, have larger farms, and rent a larger percentage of land than non-users of Extension. A higher percentage of Extension users have earned an Associate, Bachelor, or graduate degree and earn higher incomes than non-users. This is consistent with the demographic profile of producers who typically adopt precision agriculture technologies (Roberts et al., 2002; Walton et al., 2008)
Information suppliers (crop consultants, farm dealers, Extension educators and media information providers) may be able to tailor their services to clientele, based on the information provided in the study reported here. For example, because farmers tend to use Extension and private information sources simultaneously, Extension educators can tailor a more comprehensive training/outreach program for this target population in conjunction with crop consultants and/or farm dealers. This information might help to develop precision farming Extension programs that combine efforts with other sources of information to provide more effective precision farming information to consumers.
Aguilar, C., & Thornsbury, S. (2005). Limited resources—growing needs: Lessons learned in a process to facilitate program evaluation. Journal of Extension [On-line], 43(6) Article 6FEA3. Available at: http://www.joe.org/joe/2005december/a3.php
Bongiovanni, R., & Lowenberg-DeBoer, J. (2004). Precision agriculture and sustainability. Precision Agriculture, 5, 359-387.
Bullock, D. S., Lowenberg-Deboer, J., & Swinton, S. (2002). Adding value to spatially managed inputs by understanding site-specific yield response. Agricultural Economics, 27(3), 233-245.
Daberkow, S.G., & McBride, W.D. (2003). Farm and operator characteristics affecting the awareness and adoption of precision agriculture technologies in the U.S. Precision Agriculture, 4(2), 163-177.
Diem, K. G. (2002). Making program choices when resources are limited: Using a self-assessment tool with stakeholders. Journal of Extension [On-line], 40(4) Article 4TOT3. Available at: http://www.joe.org/joe/2002august/tt3.php
Griffin, T., Lowenberg-DeBoer, J., Lambert, D. M., Peone, J., Payne, T., & Daberkow, S. (2004). Precision farming: adoption, profitability, and, making better use of data. Staff Paper 06-04, Department of Agricultural Economics, Purdue University, West Lafayette, Indiana. Retrieved May 29, 2009, from: http://purl.umn.edu/28615
Hall, L., Dunkelberger, J., Ferreira, W., Prevatt, J. W., & Martin, N. R. (2003). Diffusion-adoption of personal computers and the internet in farm business decisions: Southeastern beef and peanut farmers." Journal of Extension [On-line], 41(3) Article 3FEA6. Available at: http://www.joe.org/joe/2003june/a6.php
Just, D., Wolf, S. A., Wu, S., & Zilberman, D. (2002). Consumption of economic information in agriculture. American Journal of Agricultural Economics, 84(1): 39-52.
Roberts, R. K., English, B. C., & Larson, J. A. (2002). Factors affecting the location of precision farming technology adoption in Tennessee. Journal of Extension [On-line], 40(1) Article 1RIB3. Available at: http://www.joe.org/joe/2002february/rb3.php
Schnitkey, G., Batte, M., Jones, E., & Botomogno, J. (1992). Information preferences of Ohio commercial farmers: Implications for Extension. American Journal of Agricultural Economics, 74(2): 486-496.
Smith, M. F., & Swisher, M. E. (1986). The best little programming tool in Extension. Journal of Extension [On-line], 24(3) Article 3FEA3. Available at:http://www.joe.org/joe/1986fall/a3.php
Swinton, S. M., & Lowenberg-DeBoer, J. (1998). Evaluating the profit ability of site-specific farming. Journal of Production Agriculture, 11: 439-446.
Torbett, C. J., Roberts, R. K., Larson, J. A., & English, B. C. (2007). Perceived importance of precision farming technologies in improving phosphorous and potassium efficiency in cotton production. Precision Agriculture, 8: 127-137.
Walton, J. C., Lambert, D.M., Roberts, R. K., Larson, J. A., English, B. C., Larkin, S. L., Martin, S. W., Marra, M. C., Paxton, K. W., & Reeves, J. M. (2008). Adoption and abandonment of precision soil sampling in cotton production. Journal of Agricultural and Resource Economics, 33(3): 428-448.
Watson, S., Segarra, E., Lascano, R., Bronson, K., & Schubert, A. M. (2005). Guidelines for recommending precision agriculture in southern crops. Journal of Extension [On-line], 43(2) Article 2RIB7. Available at: http://www.joe.org/joe/2005april/rb7.php


text:
\section{Introduction}
\label{sec:intro}

Large-scale educational platforms have the capability to revolutionize education by providing inexpensive, high-quality learning opportunities for millions of learners worldwide.  Examples of such platforms include massive open online courses (MOOCs) \cite{pritchardmooc,courserawebsite,khan,edxwebsite,guolick,googlemooc}, intelligent tutoring systems \cite{woolf08}, computer-based homework and testing systems \cite{erater,sapling,tesr,webassign}, and personalized learning systems \cite{ost}.  While computer and communication technologies have provided effective means to scale up the number of learners viewing lectures (via streaming video), reading the textbook (via the web), interacting with simulations (via a graphical user interface), and engaging in discussions (via online forums), the submission and grading of assessments such as homework assignments and tests remains a weak link.

There is a pressing need to find new ways and means to automate two critical tasks that are typically handled by the instructor or course assistants in a small-scale course: ({\em i})~grading of assessments, including allotting partial credit for partially correct solutions, and ({\em ii}) providing individualized feedback to learners on the locations and types of their errors.

Substantial progress has been made on automated grading and feedback systems in several restricted domains, including essay evaluation using natural language processing (NLP) \cite{erater,ease}, computer program evaluation \cite{codehint,gulwanifeedback2014,kkhint,singh2013automated,aspiringminds}, and mathematical proof verification \cite{germangrade,metamath,mizar}.

In this paper, we study the problem of automatically grading the kinds of {\em open response mathematical questions} that figure prominently in STEM (science, technology, engineering, and mathematics) education.
To the best of our knowledge, there exist no tools to automatically evaluate and allot partial-credit scores to the solutions of such questions.
As a result, large-scale education platforms have resorted either to oversimplified multiple choice input and binary grading schemes (correct/incorrect), which are known to convey less information about the learners' knowledge than open response questions \cite{ofvsmc}, or peer-grading schemes \cite{jhuangedm,joachims}, which shift the burden of grading from the course instructor to the learners.\footnote{While peer grading appears to have some pedagogical value for learners \cite{pglearning}, each learner typically needs to grade several solutions from other learners for each question they solve, in order to obtain an accurate grade estimate.}

\subsection{Main Contributions}

In this paper, we develop a data-driven framework for {\em mathematical language processing} ($\mathsf{MLP}$) that leverages solution data from a large number of learners to evaluate the correctness of solutions to open response mathematical questions, assign partial-credit scores, and provide feedback to each learner on the likely locations of any errors.
The scope of our framework is broad and covers questions whose solution involves one or more mathematical expressions.
This includes not just formal proofs but also the kinds of mathematical calculations that figure prominently in science and engineering courses.
Examples of solutions to two algebra questions of various levels of correctness are given in Figures~\ref{fig:ex1}~and~\ref{fig:ex2}.
In this regard, our work differs significantly from that of \cite{germangrade}, which focuses exclusively on evaluating logical proofs.

\begin{figure}[tp]
\vspace{-0.0cm}
\centering
\includegraphics[scale=0.35]{figs_ex1.pdf}
\caption{Example solutions to the question ``Find the derivative of $(x^3+\sin x)/e^x$'' that were assigned scores of 3, 2, 1 and 0 out of 3, respectively, by our $\mathsf{MLP\text{-}B}$ algorithm.}
\label{fig:ex1}
\end{figure}

\begin{figure}[t]
\vspace{-0.5cm}
\centering
\includegraphics[scale=0.36]{figs_ex2.pdf}
\caption{Examples of two different yet correct paths to solve the question ``Simplify the expression $(x^2+x+\sin^2 x+ \cos^2 x) (2x-3) $.''}
\label{fig:ex2}
\end{figure}

Our $\mathsf{MLP}$ framework, which is inspired by the success of NLP methods for the analysis of textual solutions (e.g., essays and short answer), comprises three main steps.

First, we convert each solution to an open response mathematical question into a series of {\em numerical features}.  In deriving these features, we make use of symbolic mathematics to transform mathematical expressions into a canonical form.

Second, we {\em cluster} the features from several solutions to uncover the structures of correct, partially correct, and incorrect solutions.
We develop two different clustering approaches.
$\mathsf{MLP\text{-}S}$ uses the numerical features to define a \textit{similarity score} between pairs of solutions and then applies a generic clustering algorithm, such as spectral clustering (SC) \cite{sc} or affinity propagation (AP) \cite{frey2007clustering}.
We show that $\mathsf{MLP\text{-}S}$ is also useful for visualizing mathematical solutions.  This can help instructors identify groups of learners that make similar errors so that instructors can deliver personalized remediation.
$\mathsf{MLP\text{-}B}$ defines a {\em nonparametric Bayesian model} for the solutions and applies a Gibbs sampling algorithm to cluster the solutions.

Third, once a human assigns a grade to at least one solution in each cluster, we automatically {\em grade} the remaining (potentially large number of) solutions based on their assigned cluster.
As a bonus, in $\mathsf{MLP\text{-}B}$, we can track the cluster assignment of each step in a multistep solution and determine when it departs from a cluster of correct solutions, which enables us to indicate the likely locations of errors to learners.

In developing $\mathsf{MLP}$, we tackle three main challenges of analyzing open response mathematical solutions.
First, solutions might contain different notations that refer to the same mathematical quantity.
For instance, in \fref{fig:ex1}, the learners use both $e^{-x}$ and $\frac{1}{e^x}$ to refer to the same quantity.
Second, some questions admit more than one path to the correct/incorrect solution.
For instance, in \fref{fig:ex2} we see two different yet correct solutions to the same question.
It is typically infeasible for an instructor to enumerate all of these possibilities to automate the grading and feedback process.
Third, numerically verifying the correctness of the solutions does not always apply to mathematical questions, especially when simplifications are required.
For example, a question that asks to simplify the expression $\sin^2 x + \cos^2 x + x$ can have both $1+x$ and $\sin^2 x + \cos^2 x + x$ as numerically correct answers, since both these expressions output the same value for all values of $x$.  However, the correct answer is $1+x$, since the question expects the learners to recognize that $\sin^2 x + \cos^2 x = 1$.
Thus, methods developed to check the correctness of computer programs and formulae by specifying a range of different inputs and checking for the correct outputs, e.g., \cite{singh2013automated}, cannot always be applied to accurately grade open response mathematical questions.

\subsection{Related Work}
\label{sec:rw}

Prior work has led to a number of methods for grading and providing feedback to the solutions of certain kinds of open response questions.
A linear regression-based approach has been developed to grade essays using features extracted from a training corpus using Natural Language Processing (NLP) \cite{erater,ease}.
Unfortunately, such a simple regression-based model does not perform well when applied to the features extracted from mathematical solutions.
Several methods have been developed for automated analysis of computer programs \cite{gulwanifeedback2014,singh2013automated}.
However, these methods do not apply to the solutions to open response mathematical questions, since they lack the structure and compilability of computer programs.
Several methods have also been developed to check the correctness of the logic in mathematical proofs \cite{germangrade,metamath,mizar}.
However, these methods apply only to mathematical proofs involving logical operations and not the kinds of open-ended mathematical calculations that are often involved in science and engineering courses.

The idea of clustering solutions to open response questions into groups of similar solutions has been used in a number of previous endeavors:
\cite{msearly,mslas} uses clustering to grade short, textual answers to simple questions;
\cite{jhuangwww} uses clustering to visualize a large collection of computer programs;
and
\cite{rivers2012canonicalizing} uses clustering to grade and provide feedback on computer programs.
Although the high-level concept underlying these works is resonant with the $\mathsf{MLP}$ framework, the feature building techniques used in $\mathsf{MLP}$ are very different, since the structure of mathematical solutions differs significantly from short textual answers and computer programs.

This paper is organized as follows.
In the next section, we develop our approach to convert open response mathematical solutions to numerical features that can be processed by machine learning algorithms.
We then develop $\mathsf{MLP\text{-}S}$ and $\mathsf{MLP\text{-}B}$ and use real-world MOOC data to showcase their ability to accurately grade a large number of solutions based on the instructor's grades for only a small number of solutions, thus substantially reducing the human effort required in large-scale educational platforms.
We close with a discussion and perspectives on future research directions.

\section{MLP Feature Extraction}
\label{sec:feature}

The first step in our $\mathsf{MLP}$ framework is to transform a collection of solutions to an open response mathematical question into a set of numerical features.
In later sections, we show how the numerical features can be used to cluster and grade solutions as well as generate informative learner feedback.

A solution to an open response mathematical question will in general contain a mixture of explanatory text and core mathematical expressions.
Since the correctness of a solution depends primarily on the mathematical expressions, we will ignore the text when deriving features.
However, we recognize that the text is potentially very useful for automatically generating explanations for various mathematical expressions.  We leave this avenue for future work.

A workhorse of NLP is the \emph{bag-of-words} model; it has found tremendous success in text semantic analysis.
This model treats a text document as a collection of words and uses the frequencies of the words as numerical features to perform tasks like topic classification and document clustering \cite{bleilda,mslas}.

A solution to an open response mathematical question consists of a series of {\em mathematical expressions} that are chained together by text, punctuation, or mathematical delimiters including $=$, $\leq$, $>$, $\propto$, $\approx$, etc.
For example, the solution in Figure~\ref{fig:ex1}(b) contains the expressions
$((x^3 + \sin x)/e^x)'$,
$((3x^2 + \cos x ) e^x - (x^3+\sin x)e^x))/e^{2x}$, and
$(2x^2 - x^3+\cos x-\sin x )/e^x$ that are all separated by the delimiter ``$=$".

$\mathsf{MLP}$ identifies the unique mathematical expressions contained in the learners' solutions and uses them as features, effectively extending the bag-of-words model to use mathematical expressions as features rather than words.
To coin a phrase, $\mathsf{MLP}$ uses a novel {\em bag-of-expressions} model.

Once the mathematical expressions have been extracted from a solution, we parse them using SymPy, the open source Python library for symbolic mathematics \cite{sympy}.\footnote{In particular, we use the {\tt parse\_expr} function.}
SymPy has powerful capability for simplifying expressions.
For example, $x^2 + x^2$ can be simplified to $2x^2$, and $e^x x^2 / e^{2x}$ can be simplifed to $e^{-x} x^2$.
In this way, we can identify the equivalent terms in expressions that refer to the same mathematical quantity, resulting in more accurate features.
In practice for some questions, however, it might be necessary to tone down the level of SymPy's simplification.
For instance, the key to solving the question in Figure~\ref{fig:ex2} is to simplify the expression using the Pythagorean identity $\sin^2 x + \cos^2 x = 1$.
If SymPy is called on to perform such a simplification automatically, then it will not be possible to verify whether a learner has correctly navigated the simplification in their solution.
For such problems, it is advisable to perform only arithmetic simplifications.

After extracting the expressions from the solutions, we transform the expressions into numerical features.
We assume that  $N$ learners submit solutions to a particular mathematical question.
Extracting the expressions from each solution using SymPy yields a total of $V$ unique expressions across the $N$ solutions.

We encode the solutions in a integer-valued {\em solution feature matrix} $\bY \in \mathbb{N}^{V \times N}$ whose rows correspond to different expressions and whose columns correspond to different solutions; that is, the $(i,j)^\text{th}$ entry of $\bY$ is given by
\begin{align*}
Y_{i,j} = \text{times expression $i$ appears in solution $j$.}
\end{align*}
Each column of $\bY$ corresponds to a numerical representation of a mathematical solution.
Note that we do not consider the ordering of the expressions in this model; such an extension is an interesting avenue for future work.
In this paper, we indicate in $\bY$ only the presence and not the frequency of an expression, i.e., $\bY \in \{0,1\}^{V \times N}$ and
\begin{align}
\label{eq:y}
Y_{i,j} = \left \{ \begin{array}{ll}
1  &\text{if} \,\, \text{expression $i$ appears in solution $j$} \\[0.1cm]
0 & \text{otherwise.}
\end{array} \right. \,
\end{align}
The extension to encoding frequencies is straightforward.

To illustrate how the matrix $\bY$ is constructed, consider the solutions in \fref{fig:ex2}(a) and (b).
Across both solutions, there are $7$ unique expressions.
Thus, $\bY$ is a $7 \times 2$ matrix, with each row corresponding to a unique expression.
Letting the first four rows of $\bY$ correspond to the four expressions in \fref{fig:ex2}(a) and the remaining three rows to expressions~2--4 in \fref{fig:ex2}(b), we have
\begin{align*}
\bY = \begin{bmatrix} 1 & 1 & 1 & 1 & 0 & 0 & 0 \\ 1 & 0 & 0 & 1 & 1 & 1 & 1 \end{bmatrix}^T.
\end{align*}

We end this section with the crucial observation that, for a wide range of mathematical questions, many expressions will be shared across learners' solutions.
This is true, for instance, in Figure~\ref{fig:ex2}.
This suggests that there are a limited number of types of solutions to a question (both correct and incorrect) and that solutions of the same type tend tend to be similar to each other.
This leads us to the conclusion that the $N$ solutions to a particular question can be effectively clustered into $K \ll N$ clusters.
In the next two sections, we will develop $\mathsf{MLP\text{-}S}$ and $\mathsf{MLP\text{-}B}$, two algorithms to cluster solutions according to their numerical features.

\begin{figure*}[!t]
\vspace{-.0cm}
\begin{center}
\includegraphics[scale=.76]{Cluster_Fig.pdf}
\end{center}
\caption{Illustration of the clusters obtained by $\mathsf{MLP\text{-}S}$ by applying affinity propagation (AP) on the similarity matrix $\bS$ corresponding to learners' solutions to four different mathematical questions (see Table~1 for more details about the datasets and the Appendix for the question statements).  Each node corresponds to a solution.  Nodes with the same color correspond to solutions that are estimated to be in the same cluster. The thickness of the edge between two solutions is proportional to their similarity score.  Boxed solutions are correct; all others are in varying degrees of correctness.
}
\label{fig:cluster}
\end{figure*}

\section{MLP-S: Similarity-Based Clustering}
\label{sec:jtree}

In this section, we outline $\mathsf{MLP\text{-}S}$, which clusters and then grades solutions using a solution similarity-based approach.

\subsection{The $\mathsf{MLP\text{-}S}$ Model}
\label{sec:metric}

We start by using the solution features in $\bY$ to define a notion of similarity between pairs of solutions.
Define the $N \times N$ {\em similarity matrix} $\bS$ containing the pairwise similarities between all solutions, with its $(i,j)^\text{th}$ entry the similarity between solutions $i$ and $j$
\begin{equation}
\label{eq:similarity}
S_{i,j} = \frac{ \vecy_i^T \vecy_j}{\min\{\vecy_i^T \vecy_i, \vecy_j^T \vecy_j \}}\,.
\end{equation}
The column vector $\vecy_i$ denotes the $i^\text{th}$ column of $\bY$ and corresponds to learner $i$'s solution.
Informally, $S_{i,j}$ is the number of common expressions between solution $i$ and solution $j$ divided by the minimum of the number of expressions in solutions $i$ and $j$.
A large/small value of $S_{i,j}$ corresponds to the two solutions being similar/dissimilar.
For example, the similarity between the solutions in Figure~\ref{fig:ex1}(a) and Figure~\ref{fig:ex1}(b) is $1/3$ and the similarity between the solutions in Figure~\ref{fig:ex2}(a) and Figure~\ref{fig:ex2}(b) is $1/2$.
$\bS$ is symmetric, and $0 \le S_{i,j} \le 1$.
Equation (\ref{eq:similarity}) is just one of any possible solution similarity metrics.
We defer the development of other metrics to future work.

\subsection{Clustering Solutions in $\mathsf{MLP\text{-}S}$}
\label{sec:cluster}

Having defined the similarity $S_{i,j}$ between two solutions $i$ and $j$, we now cluster the $N$ solutions into $K \ll N$ clusters such that the solutions within each cluster have high similarity score between them and solutions in different clusters have low similarity score between them.

Given the similarity matrix $\bS$, we can use any of the multitude of standard clustering algorithms to cluster solutions.  Two examples of clustering algorithms are {\em spectral clustering} (SC) \cite{sc} and {\em affinity propagation} (AP) \cite{frey2007clustering}.
The SC algorithm requires specifying the number of clusters $K$ as an input parameter, while the AP algorithm does not

Figure~\ref{fig:cluster} illustrates how AP is able to identify clusters of similar solutions from solutions to four different mathematical questions.  The figures on the top correspond to solutions to the questions in Figures~\ref{fig:ex1} and~\ref{fig:ex2}, respectively.  The bottom two figures correspond to solutions to two signal processing questions.  Each node in the figure corresponds to a solution, and nodes with the same color correspond to solutions that belong to the same cluster.  For each figure, we show a sample solution from some of these clusters, with the boxed solutions corresponding to correct solutions.  We can make three interesting observations from \fref{fig:cluster}:

\begin{itemize}

\item In the top left figure, we cluster a solution with the final answer $3x^2 + \cos x - (x^3 + \sin x))/e^x$ with a solution with the final answer $3^{x^2} + \cos x - (x^3+\sin x))/e^x$.  Although the later solution is incorrect, it contained a typographical error where $3 * x \wedge 2$ was typed as $3 \wedge x \wedge 2$.  $\mathsf{MLP\text{-}S}$ is able to identify this typographical error, since the expression before the final solution is contained in several other correct solutions.

\item In the top right figure, the correct solution requires identifying the trigonometric identify $\sin^2 x + \cos^2 x = 1$.  The clustering algorithm is able to identify a subset of the learners who were not able to identify this relationship and hence could not simplify their final expression.

\item $\mathsf{MLP\text{-}S}$ is able to identify solutions that are strongly connected to each other.   Such a visualization can be extremely useful for course instructors.  For example, an instructor can easily identify a group of learners who lack mastery of a certain skill that results in a common error and adjust their course plan accordingly to help these learners.

\end{itemize}

\subsection{Auto-Grading via $\mathsf{MLP\text{-}S}$}

Having clustered all solutions into a small number $K$ of clusters, we assign the same grade to all solutions in the same cluster.
If a course instructor assigns a grade to one solution from each cluster, then $\mathsf{MLP\text{-}S}$ can automatically grade the remaining $N-K$ solutions.
We construct the index set $\mathcal{I}_S$ of solutions that the course instructor needs to grade as
\begin{align*}
\mathcal{I}_S = \left\{\arg \max_{i \in {\cal C}_k} \sum_{j = 1}^N S_{i,j}, \, k = 1,2,\ldots, K \right\},
\end{align*}
where ${\cal C}_k$ represents the index set of the solutions in cluster $k$.
In words, in each cluster, we select the solution having the highest similarity to the other solutions (ties are broken randomly) to include in $\mathcal{I}_S$.  We demonstrate the performance of auto-grading via $\mathsf{MLP\text{-}S}$ in the experimental results section below.

\section{MLP-B: Bayesian nonparametric clustering}
\label{sec:crp}

\fussy

In this section, we outline $\mathsf{MLP\text{-}B}$, which clusters and then grades solutions using a Bayesian nonparameterics-based approach.
The $\mathsf{MLP\text{-}B}$ model and algorithm can be interpreted as an extension of the model in \cite{shorttext}, where a similar approach is proposed to cluster short text documents.

\subsection{The $\mathsf{MLP\text{-}B}$ Model}
\label{sec:model}

Following the key observation that the $N$ solutions can be effectively clustered into $K \ll N$ clusters, let $\vecz$ be the $N \times 1$ cluster assignment vector, with $z_j \in \{1,\ldots,K\}$ denoting the cluster assignment of the $j^\text{th}$ solution with $j \in \{1,\ldots,N\}$. Using this latent variable, we model the probability of the solution of all learners' solutions to the question as
\begin{align*}
p(\bY) = \prod_{j=1}^N \left( \sum_{k=1}^K p(\vecy_j | z_j = k) p(z_j = k) \right),
\end{align*}
where $\vecy_j$, the $j^\text{th}$ column of the data matrix $\bY$, corresponds to learner $j$'s solution to the question. Here we have implicitly assumed that the learners' solutions are independent of each other.
By analogy to topic models \cite{bleilda,tm}, we assume that learner $j$'s solution to the question, $\vecy_j$, is generated according to a \emph{multinomial} distribution given the cluster assignments $\vecz$ as
 \begin{align} \label{eq:lik}
\notag p(\vecy_j | z_j = k) & = \textit{Mult}(\vecy_j | \boldsymbol{\phi}_k) \\
& = \frac{ (\sum_i Y_{i,j})!}{Y_{1,j}! Y_{2,j}! \ldots Y_{V,j}!} \Phi_{1,k}^{Y_{1,j}} \Phi_{2,k}^{Y_{2,j}} \ldots \Phi_{V,k}^{Y_{V,j}},
\end{align}
where $\boldsymbol{\Phi} \in [0,1]^{V \times K}$ is a parameter matrix with $\Phi_{v,k}$ denoting its $(v,k)^\text{th}$ entry.
$\boldsymbol{\phi}_k \in [0,1]^{V \times 1}$ denotes the $k^\text{th}$ column of $\boldsymbol{\Phi}$ and charcterizes the multinomial distribution over all the $V$ features for cluster $k$.

In practice, one often has no information regarding the number of clusters $K$. Therefore, we consider $K$ as an unknown parameter and infer it from the solution data. In order to do so, we impose a Chinese restaurant process (CRP) prior on the cluster assignments $\vecz$, parameterized by a parameter $\alpha$. The CRP characterizes the random partition of data into clusters, in analogy to the seating process of customers in a Chinese restaurant.
It is widely used in Bayesian mixture modeling literature \cite{crptm,crp}. Under the CRP prior, the cluster (table) assignment of the $j^\text{th}$ solution (customer), conditioned on the cluster assignments of all the other solutions, follows the distribution
\begin{align} \label{eq:crp}
p(z_j = k | \vecz_{\neg j}, \alpha) = \left \{ \begin{array}{ll}
\frac{n_{k,\neg j}}{N-1+\alpha}  & \text{if} \,\,\, \text{cluster $k$ is occupied}, \\[0.1cm]
\frac{\alpha}{N-1+\alpha} & \text{if} \,\,\, \text{cluster $k$ is empty},
\end{array} \right.
\end{align}
where $n_{k,\neg j}$ represents the number of solutions that belong to cluster $k$ excluding the current solution $j$, with $\sum_{k=1}^K n_{k,\neg j} = N-1$.  The vector $\vecz_{\neg j}$ represents the cluster assignments of the other solutions.  The flexibility of allowing any solution to start a new cluster of its own enables us to automatically infer $K$ from data. It is known \cite{tehdirichlet} that the expected number of clusters under the CRP prior satisfies $K \sim O(\alpha \, \text{log}N) \ll N$, so our method scales well as the number of learners $N$ grows large. We also impose a Gamma prior $\alpha \sim \textit{Gam}(\alpha_\alpha, \alpha_\beta)$ on $\alpha$ to help us infer its value.

Since the solution feature data $\bY$ is assumed to follow a multinomial distribution parameterized by $\boldsymbol{\Phi}$, we impose a symmetric Dirichlet prior over $\boldsymbol{\Phi}$ as $\boldsymbol{\phi}_k \sim \textit{Dir} (\boldsymbol{\phi}_k | \beta)$ because of its conjugacy with the multinomial distribution \cite{gelmanbook}.

\begin{figure}[tp]
\vspace{-0.2cm}
\centering
\includegraphics[scale=0.19]{plate.pdf}
\vspace{.0cm}
\caption{Graphical model of the generation process of solutions to mathematical questions. $\alpha_\alpha$, $\alpha_\beta$ and $\beta$ are hyperparameters, $\vecz$ and $\boldsymbol{\Phi}$ are latent variables to be inferred, and $\bY$ is the observed data defined in \fref{eq:y}.}
\label{fig:plate}
\vspace{-0.0cm}
\end{figure}

The graphical model representation of our model is visualized in \fref{fig:plate}.
Our goal next is to estimate the cluster assignments $\vecz$ for the solution of each learner, the parameters $\boldsymbol{\phi}_k$ of each cluster, and the number of clusters $K$, from the binary-valued solution feature data matrix $\bY$.

\subsection{Clustering Solutions in $\mathsf{MLP\text{-}B}$}
\label{sec:inference}

We use a Gibbs sampling algorithm for posterior inference under the $\mathsf{MLP\text{-}B}$ model, which automatically groups solutions into clusters.  We start by applying a generic clustering algorithm (e.g., $K$-means, with $K = N/10$) to initialize $\vecz$, and then initialize $\boldsymbol{\Phi}$ accordingly. Then, in each iteration of $\mathsf{MLP\text{-}B}$, we perform the following steps:
\begin{itemize}

\item[1.] {\bf Sample $\vecz$}: For each solution $j$, we remove it from its current cluster and sample its cluster assignment $z_j$ from the posterior $p(z_j = k | \vecz_{\neg j}, \alpha, \bY)$.  Using Bayes rule, we have
\begin{align*}
  p(z_j = k | \vecz_{\neg j}, \boldsymbol{\Phi}, \alpha, \bY)
& = p(z_j = k | \vecz_{\neg j}, \boldsymbol{\phi}_k, \alpha, \vecy_j) \\
& \propto \! p( \! z_j \! = \! k | \vecz_{\neg j}, \! \alpha) p ( \vecy_j | z_j \! = \! k, \! \boldsymbol{\phi}_k \!).
\end{align*}
The prior probability $p(z_j = k | \vecz_{\neg j}, \alpha)$ is given by \fref{eq:crp}. For non-empty clusters, the observed data likelihood $p ( \vecy_j | z_j = k, \boldsymbol{\phi}_k)$ is given by \fref{eq:lik}. However, this does not apply to new clusters that are previously empty. For a new cluster, we marginalize out $\boldsymbol{\phi}_k$, resulting in
\begin{align*}
p(\vecy_j | z_j = k, \beta) & = \int_{\boldsymbol{\phi}_k} p(\vecy_j | z_j = k, \boldsymbol{\phi}_k) p(\boldsymbol{\phi}_k | \beta) \\
& = \int_{\boldsymbol{\phi}_k} \textit{Mult}(\vecy_j | z_j = k,\boldsymbol{\phi}_k ) \textit{Dir}(\boldsymbol{\phi}_k | \beta) \\
& = \frac{\Gamma(V \beta)}{\Gamma( \sum_{i=1}^V Y_{i,j} + V \beta)} \prod_{i=1}^V \frac{\Gamma(Y_{i,j} + \beta)}{\Gamma(\beta)},
\end{align*}
where $\Gamma(\cdot)$ is the Gamma function.

If a cluster becomes empty after we remove a solution from its current cluster, then  we remove it from our sampling process and erase its corresponding multinomial parameter vector $\boldsymbol{\phi}_k$.
If a new cluster is sampled for $z_j$, then we sample its multinomial parameter vector $\boldsymbol{\phi}_k$ immediately according to Step~2 below. Otherwise, we do not change $\boldsymbol{\phi}_k$ until we have finished sampling $\vecz$ for all solutions.

\item[2.] {\bf Sample $\boldsymbol{\Phi}$}: For each cluster $k$, sample $\boldsymbol{\phi}_k$ from its posterior $\textit{Dir}(\boldsymbol{\phi}_k | n_{1,k} + \beta, \ldots, n_{V,k} + \beta)$, where $n_{i,k}$ is the number of times feature $i$ occurs in the solutions that belong to cluster $k$.

\item[3.] {\bf Sample $\alpha$}: Sample $\alpha$ using the approach described in \cite{westparam}.

\item[4.] {\bf Update $\beta$}: Update $\beta$ using the fixed-point procedure described in \cite{minkadirichlet}.
\end{itemize}

\sloppy
The output of the Gibbs sampler is a series of samples that correspond to the approximate posterior distribution of the various parameters of interest.  To make meaningful inference for these parameters (such as the posterior mean of a parameter), it is important to appropriately post-process these samples.  For our estimate of the true number of clusters, $\widehat{K}$, we simply take the mode of the posterior distribution on the number of clusters $K$. We use only iterations with $K = \widehat{K}$ to estimate the posterior statistics \cite{nonparamsparfab}.

In mixture models, the issue of ``label-switching'' can cause a model to be unidentifiable, because the cluster labels can be arbitrarily permuted without affecting the data likelihood. In order to overcome this issue, we use an approach reported in \cite{nonparamsparfab}. First, we compute the likelihood of the observed data in each iteration as $p(\bY | \boldsymbol{\Phi}^\ell, \vecz^\ell)$, where $\boldsymbol{\Phi}^\ell$ and $\vecz^\ell$ represent the samples of these variables at the $\ell^\text{th}$ iteration. After the algorithm terminates, we search for the iteration $\ell_\text{max}$ with the largest data likelihood and then permute the labels $\vecz^\ell$ in the other iterations to best match $\boldsymbol{\Phi}^{\ell}$ with $\boldsymbol{\Phi}^{\ell_\text{max}}$.
We use $\widehat{\boldsymbol{\Phi}}$ (with columns $\hat{\boldsymbol{\phi}}_k$) to denote the estimate of $\boldsymbol{\Phi}$, which is simply the posterior mean of $\boldsymbol{\Phi}$.
Each solution $j$ is assigned to the cluster indexed by the mode of the samples from the posterior of $z_j$, denoted by $\hat{z}_j$.

\subsection{Auto-Grading via $\mathsf{MLP\text{-}B}$}
\label{sec:auto}

We now detail how to use $\mathsf{MLP\text{-}B}$ to automatically grade a large number $N$ of learners' solutions to a mathematical question, using a small number $\widehat{K}$ of instructor graded solutions.
First, as in $\mathsf{MLP\text{-}S}$, we select the set $\mathcal{I}_B$ of ``typical solutions'' for the instructor to grade.
We construct $\mathcal{I}_B$ by selecting one solution from each of the $\widehat{K}$ clusters that is most representative of the solutions in that cluster:
\begin{align*}
\mathcal{I}_B = \{ \arg \max_j p(\vecy_j | \hat{\boldsymbol{\phi}}_k), k = 1,2,\ldots,\widehat{K} \}.
\end{align*}
In words, for each cluster, we select the solution with the largest likelihood of being in that cluster.

The instructor grades the $\widehat{K}$ solutions in $\mathcal{I}_B$ to form the set of instructor grades $\{ g_k \}$ for $k \in \mathcal{I}_B$.
Using these grades, we assign grades to the other solutions $j \notin \mathcal{I}_B$ according to
\begin{align} \label{eq:grade}
\hat{g}_j = \frac{\sum_{k=1}^{\widehat{K}} p(\vecy_j | \hat{\boldsymbol{\phi}}_k) g_k}{\sum_{k=1}^{\widehat{K}} p(\vecy_j | \hat{\boldsymbol{\phi}}_k)}.
\end{align}
That is, we grade each solution not in $\mathcal{I}_B$ as the average of the instructor grades weighted by the likelihood that the solution belongs to cluster.
We demonstrate the performance of auto-grading via $\mathsf{MLP\text{-}B}$ in the experimental results section below.

\subsection{Feedback Generation via $\mathsf{MLP\text{-}B}$}
\label{sec:alert}

In addition to grading solutions, $\mathsf{MLP\text{-}B}$ can automatically provide useful feedback to learners on where they made errors in their solutions.

For a particular solution $j$ denoted by its column feature value vector $\vecy_j$ with $V_j$ total expressions, let $\vecy_j^{(v)}$ denote the feature value vector that corresponds to the first $v$ expressions of this solution, with $v = \{1,2,\ldots,V_j\}$.
Under this notation, we evaluate the probability that the first $v$ expressions of solution $j$ belong to each of the $\widehat{K}$ clusters: $p(\vecy_j^{(v)} | \hat{\boldsymbol{\phi}}_k), k = \{1,2,\ldots,\widehat{K}\}$, for all $v$.
Using these probabilities, we can also compute the expected credit of solution $j$ after the first $v$ expressions via
\begin{align} \label{eq:expgrade}
\hat{g}_j^{(v)} = \frac{\sum_{k=1}^{\widehat{K}} p(\vecy_j^{(v)} | \hat{\boldsymbol{\phi}}_k) g_k}{\sum_{k=1}^{\widehat{K}} p(\vecy_j^{(v)} | \hat{\boldsymbol{\phi}}_k)},
\end{align}
where $\{g_k\}$ is the set of instructor grades as defined above.

Using these quantities, it is possible to identify that the learner has likely made an error at the $v^\text{th}$ expression if it is most likely to belong to a cluster with credit $g_k$ less than the full credit or, alternatively, if the expected credit $\hat{g}_j^{(v)}$ is less than the full credit.

The ability to automatically locate {\em where} an error has been made in a particular incorrect solution provides many benefits.  For instance, $\mathsf{MLP\text{-}B}$ can inform instructors of the most common locations of learner errors to help guide their instruction.  It can also enable an automated tutoring system to generate feedback to a learner as they make an error in the early steps of a solution, before it propagates to later steps.  We demonstrate the efficacy of $\mathsf{MLP\text{-}B}$ to automatically locate learner errors using real-world educational data in the experiments section below.

\section{Experiments}
\label{sec:sims}

In this section, we demonstrate how $\mathsf{MLP\text{-}S}$ and $\mathsf{MLP\text{-}B}$ can be used to accurately estimate the grades of roughly $100$ open response solutions to mathematical questions by only asking the course instructor to grade approximately $10$ solutions.  We also demonstrate how $\mathsf{MLP\text{-}B}$ can be used to automatically provide feedback to learners on the locations of errors in their solutions.

\subsection{Auto-Grading via $\mathsf{MLP\text{-}S}$ and $\mathsf{MLP\text{-}B}$}

\paragraph{Datasets}
Our dataset that consists of $116$ learners solving $4$ open response mathematical questions in an edX course.  The set of questions includes $2$ high-school level mathematical questions and $2$ college-level signal processing questions (details about the questions can be found in \fref{tbl:dataset}, and the question statements are given in the Appendix).  For each question, we pre-process the solutions to filter out the blank solutions and extract features.   Using the features, we represent the solutions by the matrix $\bY$ in \fref{eq:y}.
Every solution was graded by the course instructor with one of the scores in the set $\{0,1,2,3\}$, with a full credit of $3$.

\begin{table}
\vspace{-0.0cm}
\centering
\caption{Datasets consisting of the solutions of $116$ learners to $4$ mathematical questions on algebra and signal processing.  See the Appendix for the question statements.}
\vspace{-0.0cm}
\label{tbl:dataset}
\scalebox{.77}{
    \begin{tabular}{ccc}
\toprule
   &  No.of solutions $N$ & No.of features (unique expressions) $V$ \\
\midrule
Question~1 & 108 & 78  \\
Question~2 & 113 & 53   \\
Question~3 & 90 & 100   \\
Question~4 & 110 & 45   \\
\bottomrule
\end{tabular}}
\vspace{-0.0cm}
\end{table}

\paragraph{Baseline: Random sub-sampling}
We compare the auto-grading performance of $\mathsf{MLP\text{-}S}$ and $\mathsf{MLP\text{-}B}$ against a baseline method that does not group the solutions into clusters.  In this method, we randomly sub-sample all solutions to form a small set of solutions for the instructor to grade.  Then, each ungraded solution is simply assigned the grade of the solution in the set of instructor-graded solutions that is most similar to it as defined by $\bS$ in \fref{eq:similarity}.
Since this small set is picked randomly, we run the baseline method $10$ times and report the best performance.\footnote{Other baseline methods, such as the linear regression-based method used in the edX essay grading system \cite{ease}, are not listed, because they did not perform as well as random sub-sampling in our experiments.}

\paragraph{Experimental setup}

For each question, we apply four different methods for auto-grading:

\begin{itemize}

\item Random sub-sampling (RS) with the number of clusters $K \in \{ 5, 6, \dots, 40\}$.

\item $\mathsf{MLP\text{-}S}$ with spectral clustering (SC) with $K \in \{ 5, 6, \dots, 40\}$.

\item$\mathsf{MLP\text{-}S}$ with affinity propagation (AP) clustering.  This algorithm does not require $K$ as an input.

\item $\mathsf{MLP\text{-}B}$ with hyperparameters set to the non-informative values $\alpha_\alpha = \alpha_\beta = 1$ and running the Gibbs sampling algorithm for 10,000 iterations with 2,000 burn-in iterations.

\end{itemize}
$\mathsf{MLP\text{-}S}$ with AP and $\mathsf{MLP\text{-}B}$ both automatically estimate the number of clusters $K$.
Once the clusters are selected, we assign one solution from each cluster to be graded by the instructor using the methods described in earlier sections.

\paragraph{Performance metric}

We use mean absolute error (MAE), which measures the ``average absolute error per auto-graded solution''
\begin{align*}
\text{MAE} = \frac{\sum_{j=1}^{N-K} |\hat{g}_j - g_j |}{N-K},
\end{align*}
\sloppy
as our performance metric.
Here, $N-K$ equals the number of solutions that are auto-graded, and $\hat{g}_j$ and $g_j$ represent the estimated grade (for $\mathsf{MLP\text{-}B}$, the estimated grades are rounded to integers) and the actual instructor grades for the auto-graded solutions, respectively.

\paragraph{Results and discussion}

In \fref{fig:errorvsk}, we plot the MAE versus the number of clusters $K$ for Questions~1--4.  $\mathsf{MLP\text{-}S}$ with SC consistently outperforms the random sampling baseline algorithm for almost all values of $K$.  This performance gain is likely due to the fact that the baseline method does not cluster the solutions and thus does not select a good subset of solutions for the instructor to grade.  $\mathsf{MLP\text{-}B}$ is more accurate than $\mathsf{MLP\text{-}S}$ with both SC and AP and can automatically estimate the value of $K$, although at the price of significantly higher computational complexity (e.g., clustering and auto-grading one question takes $2$ minutes for $\mathsf{MLP\text{-}B}$ compared to only $5$ seconds for $\mathsf{MLP\text{-}S}$ with AP on a standard laptop computer with a 2.8GHz CPU and 8GB memory).

Both $\mathsf{MLP\text{-}S}$ and $\mathsf{MLP\text{-}B}$ grade the learners' solutions accurately (e.g., an MAE of $0.04$ out of the full grade $3$ using only $K=13$ instructor grades to auto-grade all $N=113$ solutions to Question~2).  Moreover, as we see in \fref{fig:errorvsk}, the MAE for $\mathsf{MLP\text{-}S}$ decreases as $K$ increases, and eventually reaches $0$ when $K$ is large enough that only solutions that are exactly the same as each other belong to the same cluster.  In practice, one can tune the value of $K$ to achieve a balance between maximizing grading accuracy and minimizing human effort.  Such a tuning process is not necessary for $\mathsf{MLP\text{-}B}$, since it automatically estimates the value of $K$ and achieves such a balance.

\begin{figure}[t]
\vspace{-0.1cm}
\centering
\hspace{-.7cm}
\begin{subfigure}[t]{0.4\linewidth}
\includegraphics[scale=0.22]{q1err.pdf}
\subcaption{Question~1}
\end{subfigure}
\hspace{.7cm}
\begin{subfigure}[t]{0.4\linewidth}
\includegraphics[scale=0.22]{q2err.pdf}
\subcaption{Question~2}
\end{subfigure}
\\
\hspace{-.7cm}
\begin{subfigure}[t]{0.4\linewidth}
\includegraphics[scale=0.22]{q3err.pdf}
\subcaption{Question~3}
\end{subfigure}
\hspace{.7cm}
\begin{subfigure}[t]{0.4\linewidth}
\includegraphics[scale=0.22]{q4err.pdf}
\subcaption{Question~4}
\end{subfigure}
\caption{Mean absolute error (MAE) versus the number of instructor graded solutions (clusters) $K$, for Questions~1--4, respectively.  For example, on Question~1, $\mathsf{MLP\text{-}S}$ and $\mathsf{MLP\text{-}B}$ estimate the true grade of each solution with an average error of around $0.1$ out of a full credit of $3$.  ``RS'' represents the random sub-sampling baseline. Both $\mathsf{MLP\text{-}S}$ methods and $\mathsf{MLP\text{-}B}$ outperforms the baseline method.}
\label{fig:errorvsk}
\end{figure}

\subsection{Feedback Generation via $\mathsf{MLP\text{-}B}$}

\paragraph{Experimental setup}

Since Questions~3--4 require some familiarity with signal processing, we demonstrate the efficacy of $\mathsf{MLP\text{-}B}$ in providing feedback on mathematical solutions on Questions~1--2.
Among the solutions to each question, there are a few types of common errors that more than one learner makes.
We take one incorrect solution out of each type and run $\mathsf{MLP\text{-}B}$ on the other solutions to estimate the parameter $\hat{\boldsymbol{\phi}}_k$ for each cluster.
Using this information and the instructor grades $\{ g_k \}$, after each expression $v$ in a solution, we compute the probability that it belongs to a cluster $p(\vecy_j^{(v)} | \hat{\boldsymbol{\phi}}_k)$ that does not have full credit ($g_k < 3$), together with the expected credit using \fref{eq:expgrade}.  Once the expected grade is calculated to be less than full credit, we consider that an error has occurred.

\paragraph{Results and discussion}
Two sample feedback generation process are shown in \fref{fig:alert}.
In \fref{fig:alert}(a), we can provide feedback to the learner on their error as early as Line~2, before it carries over to later lines.
Thus, $\mathsf{MLP\text{-}B}$ can potentially become a powerful tool to generate timely feedback to learners as they are solving mathematical questions, by analyzing the solutions it gathers from other learners.

\begin{figure}[t]
\vspace{-0.1cm}
\centering
\scalebox{.9}{
\begin{subfigure}[t]{.9\linewidth}
\centering
\[ \begin{aligned}
& ((x^3 + \sin x) / e^x)' \\
& = ( e^x (x^3+\sin x)' - (x^3 + \sin x) (e^x)' )/e^{2x} \\
& \mathsf{prob. incorrect} = {\color{dgreen}0.11}, \quad \mathsf{exp. grade} = \color{dgreen}3\\
& = ( e^x (2x^2+\cos x) - (x^3 + \sin x) e^x )/e^{2x} \\
& \mathsf{prob. incorrect} = {\color{orange}0.66}, \quad \mathsf{exp. grade} = \color{orange}2\\
& = ( 2x^2 + \cos x - x^3 - \sin x) / e^x \\
& \mathsf{prob. incorrect} = {\color{red}0.93}, \quad \mathsf{exp. grade} = \color{orange}2 \\
& = ( x^2 (2-x) + \cos x - \sin x) / e^x \\
& \mathsf{prob. incorrect} = {\color{red}0.99}, \quad \mathsf{exp. grade} = \color{orange}2
\end{aligned} \]
\subcaption{A sample feedback generation process where the learner makes an error in the expression in Line~2 while attempting to solve Question~1.}
\label{fig:alert_a}
\end{subfigure} }\\
\scalebox{.9}{
\begin{subfigure}[t]{.9\linewidth}
\centering
\[ \begin{aligned}
& (x^2 + x + \sin^2 x + \cos^2 x) (2x-3) \\
& = (x^2 + x + 1) (2x-3)  \\
& \mathsf{prob. incorrect} = {\color{dgreen}0.09}, \quad \mathsf{exp. grade} = \color{dgreen}3\\
& = 4x^3 + 2x^2 + 2x - 3x^2 - 3x - 3 \\
& \mathsf{prob. incorrect} = {\color{orange}0.82}, \quad \mathsf{exp. grade} = \color{orange}2\\
& = 4x^3 - x^2 - x - 3 \\
& \mathsf{prob. incorrect} = {\color{red}0.99}, \quad \mathsf{exp. grade} = \color{orange}2\\
\end{aligned} \]
\subcaption{A sample feedback generation process where the learner makes an error in the expression in Line~3 while attempting to solve Question~2.}
\end{subfigure}}
\caption{Demonstration of real-time feedback generation by $\mathsf{MLP\text{-}B}$ while learners enter their solutions.
After each expression, we compute both the probability that the learner's solution belongs to a cluster that does not have full credit and the learner's expected grade.  An alert is generated when the expected credit is less than full credit.}
\label{fig:alert}
\end{figure}

\section{Conclusions}
\label{sec:conclusions}

We have developed a framework for mathematical language processing $(\mathsf{MLP})$ that consists of three main steps:
({\em i}) converting each solution to an open response mathematical question into a series of numerical features;
({\em ii}) clustering the features from several solutions to uncover the structures of correct, partially correct, and incorrect solutions; and
({\em iii}) automatically grading the remaining (potentially large number of) solutions based on their assigned cluster and one instructor-provided grade per cluster.
As our experiments have indicated, our framework can substantially reduce the human effort required for grading in large-scale courses.
As a bonus, $\mathsf{MLP\text{-}S}$ enables instructors to visualize the clusters of solutions to help them identify common errors and thus groups of learners having the same misconceptions.
As a further bonus, $\mathsf{MLP\text{-}B}$  can track the cluster assignment of each step of a multistep solution and determine when it departs from a cluster of correct solutions, which enables us to indicate the locations of errors to learners in real time.
Improved learning outcomes should result from these innovations.

There are several avenues for continued research.
We are currently planning more extensive experiments on the edX platform involving tens of thousands of learners.
We are also planning to extend the feature extraction step to take into account both the ordering of expressions and ancillary text in a solution.
Clustering algorithms that allow a solution to belong to more than one cluster could make $\mathsf{MLP}$ more robust to outlier solutions and further reduce the number of solutions that the instructors need to grade.
Finally, it would be interesting to explore how the features of solutions could be used to build predictive models, as in the Rasch model \cite{rasch} or item response theory \cite{lordirt}.

\section{Appendix: Question Statements}

Question~1:
Multiply \[(x^2+x+\sin^2 x + \cos^2 x)(2x-3) \]
and simplify your answer as much as possible.

Question~2: Find the derivative of $\displaystyle{\frac{x^3 + \sin(x)}{e^x}}$ and simplify your answer as much as possible.

Question~3:
A discrete-time linear time-invariant system has the impulse response shown in the figure (omitted). Calculate $H(e^{j\omega})$, the discrete-time Fourier transform of $h[n]$. Simplify your answer as much as possible until it has no summations.

Question~4: Evaluate the following summation
\[ \sum\limits_{k=-\infty}^{\infty}\delta[n-k] \, x[k-n]. \]

\section*{Acknowledgments}

\sloppy
Thanks to Heather Seeba for administering the data collection process and Christoph Studer for discussions and insights.  Visit our website \url{www.sparfa.com}, where you can learn more about our project and purchase t-shirts and other merchandise.

\balance

\bibliographystyle{acm-sigchi}

redpajama_set_name: RedPajamaArXiv


text:
Research Factor is Journal Impact Factor
Send journals for indexing to journal@researchfactor.org .
Research Factor is indexing service and provides impact factor of journals.
Get your indexed by submitting journals to journal@researchfactor.org.
Get Impact Factor of Journals
Send journals for indexing to journal@researchfactor.org
Impact Factor is now Research Factor and journal is indexed in collaboration with Eduindex.
Get Indexed in Research Factor
Send journals to journal@researchfactor.org for indexing and impact factor.
By EduINDEX - Indexing of Academic Contents January 04, 2020 Impact-Factor, Multidisciplinary-Journals
Details of Journal
Name of Journal
Short Form of Journal
SIPN
ISSN no.
Logo of Journal
URL of journal
https://archives.tpnsindia.org/index.php/sipn
Brief of journal
Studies in Indian Place Names (SIPN) with ISSN 2394-3114 is UGC Care listed journal for research publication. SIPN considers review and research articles related to Social Science and Humanities: Arts and Humanities, Physical Education, Library Science, History, Anthropology, Management, Commerce, Home Science, Sociology, Hotel Management, Tourism, Mass Communication, Accounting, Education, Economics, Law, Philosophy, Finance, Political Science, Visual Arts, Performing Arts, English. Science: Engineering (All Branches), Psychology, and Architecture, Geography and Geology, Agricultural, Biological Sciences, Environmental Science, Ecology, Archaeology, Biodiversity and Conservation, Entomology, Health Science: Medicine and Dentistry, Nursing and Allied Health Science, Ayurveda.
On behalf of Studies in Indian Place Names, I would like to extend my regard to all fellow researchers and scholars and wish prosperity in their field. Published by: The Place Names Society of India, NPS India
Send papers for publication to editor@tpnsindia.org
Stating year
No. of Issues per Vol.
No. of Volume per year
Multidisciplinary Journal
Year of Assessment
Our Heritage Journal - UGC Care listed
By EduINDEX - Indexing of Academic Contents January 04, 2020 Multidisciplinary-Journals, UGC-Approved-Journals, UGC-Care-Listed
https://archives.ourheritagejournal.com/index.php/oh
Our Heritage with ISSN 0474-9030 is a UGC CARE Journal and peer-reviewed open access journal published monthly in English-language only. OUR HERITAGE is a multidisciplinary journal dedicated to the research publication in the fields of architecture, business, commerce, development studies, economics, finance, government policies, history, international relations, jurisprudence, knowledgebase, learning, management studies, novel and book reviews, organizational studies, poetry, quasi-judicial studies, resource management, social studies, temporal studies, universal laws, venture capitals, word economics, youth affairs, and allied fields. , Our Heritage publishes a wide range of researches and studies on diverse fields of knowledge. OUR HERITAGE Journal gives priority to the latest findings and achievements from experts and scholars all over the world. Each issue contains a variety of critical articles, extensive reviews articles and so on.

redpajama_set_name: RedPajamaCommonCrawl


text:
Q: Pythonic way of summing lists and lists of lists I'm trying to find a neat way of summing a list and a list of lists in the same function, so far I've got:
import operator
"""
 Fails late for item = ['a', 'b']
"""
def validate(item):
    try:
        return sum(item) == sum(range(1, 10))
    except TypeError:
        return sum(reduce(operator.add, item)) == sum(range(1, 10))

"""
 Not valid for item = [1,2,[3,4,5]]
"""
def validate2(item):
        if isinstance(item[0], int):
            return sum(item) == sum(range(1, 10))
        else:
            return sum(reduce(operator.add, item)) == sum(range(1, 10))

print validate([1, 2, 3, 4, 5, 6, 7, 8, 9])
print validate([[1, 2, 3], [4, 5, 6], [7, 8, 9]])

print validate2([1, 2, 3, 4, 5, 6, 7, 8, 9])
print validate2([[1, 2, 3], [4, 5, 6], [7, 8, 9]])

...but neither of these seem quite right to me (reasons in the doc strings). What I want to know is if there is a better way of summing lists and lists of lists that doesn't require me to catch exceptions or actually analyse the list before the function decides what to do.
Obviously, I'm still expecting ['a', 'b'] to be invalid.

A: Perhaps you'd find it easier to flatten the list first?
def flatten(xs):
     for x in xs:
        try:
            sub = iter(x)
        except TypeError:
            yield x
        else:
            for y in flatten(sub):
                yield y

With the above, you can do this:
In [4]: fs = flatten([1,2,[3,4,[5,6],7],8,[9,10]])

In [5]: list(fs)
Out[5]: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

A: Don't forget to describe exactly what you're trying to do.  I'm assuming you mean to sum all values to a single value, and not to get eg. [[1,2],[3,4]] -> [3,7].  Here's simple recursion; five lines of code if you skip the tests:
def sums(it):
    """
    >>> sums(1)
    1
    >>> sums([1,2,3])
    6
    >>> sums([1,2,3,[4,5]])
    15
    >>> sums(['a','b'])
    Traceback (most recent call last):
    ...
    TypeError: unsupported operand type(s) for +: 'int' and 'str'
    """
    if getattr(it, "__iter__", None):
        return sum(map(sums, it))
    else:
        return it

if __name__ == "__main__":
    import doctest
    doctest.testmod()

A: The external numpy module has many operations (including sum()) which work similarly on scalars, vectors, matrices and even higher-dimensional arrays...
Note however that it doesn't work on mixed lists like [1,2,[3,4,5]], only square matrices!  So it doesn't exactly answer your question.

redpajama_set_name: RedPajamaStackExchange


There's no denying it, there's magic in the air.
The festive season is more than a spectacle, it's a feeling — that fuzzy glow as friends and family gather together, those sparkles in our eyes as we celebrate each other and the season. Our holiday bouquet collection, THE GIFT OF STYLE, is inspired by the joyful energy around us and winter wildflowers. Bathed in vivacious violets, rich reds, and champagne-pinks, our flowers reimagine — and redefine — festive style.
Balancing style and glamour with warmth and comfort, THE GIFT OF STYLE is the perfect amount of chic festivity to make every occasion at home special — from family feasts to glitzy get-togethers.
In pursuit of captivating bouquets that tell a story of the season, our holiday collection combines a sumptuous colour palette, iconic Christmas stems, and striking winter flowers for you to discover. Look forward to stems like cascading euphorbia fulgens, velveteen roses, and decadent variegated chrysanthemums in our bouquets, combined with the glistening berries on branches of Ilex verticillata, trumpet-shaped amaryllis, and incandescent anemones.
Fragrance is at the heart of this collection — a powerful sense that triggers memories when you interact with nostalgic scents. Transport yourself to your happy place with fragrant blooms like lemony-fresh waxflowers, camphoraceous eucalyptus, fresh-cut pine, and Matthiola — a scent so deliciously festive, it's the top note in our scented candle.
The most magical time of the year is when our home really shines. All dressed up for the festive season in wreaths, candles and, of course, the tree — it's where the celebrations take place.
Whilst key colours like rich reds, evergreens, and topaz-gold are in abundance in our holiday bouquet collection, expect to be delighted by accents of modern festive colours like champagne, blush-pink, maroon, and violet.
The modern way to dress your home for the festive season starts with flowers that shimmer in festive colours. Our new ceramic vase is designed to complement the sumptuous colours of the holiday collection. Whether you dot mini vases of single stems throughout your space, welcome in good tidings with a hand-made floral wreath, or have a bouquet as the guest of honour at your table, fresh flowers usher in a contemporary festive style by the sleigh load.
Expect the unexpected in THE GIFT OF STYLE. bloomon's holiday bouquet collection is here to celebrate the season with you.


50%55°37°Partly cloudy with a high of 55 °F (12.8 °C) and a 50% chance of precipitation. Winds variable.Night - Mostly clear. Winds variable at 2 to 7 mph (3.2 to 11.3 kph). The overnight low will be 37 °F (2.8 °C).
40%64°41°Rain and snow today with a high of 64 °F (17.8 °C) and a low of 41 °F (5.0 °C). There is a 40% chance of precipitation.
40%67°38°Rain and snow today with a high of 67 °F (19.4 °C) and a low of 38 °F (3.3 °C). There is a 40% chance of precipitation.
Today - Partly cloudy with a high of 55 °F (12.8 °C) and a 50% chance of precipitation. Winds variable.
Tonight - Mostly clear. Winds variable at 2 to 7 mph (3.2 to 11.3 kph). The overnight low will be 37 °F (2.8 °C).


text:
Homalocnemis praesumpta är en tvåvingeart som beskrevs av Collin 1933. Homalocnemis praesumpta ingår i släktet Homalocnemis och familjen Homalocnemiidae. Inga underarter finns listade i Catalogue of Life.

Källor

Tvåvingar
praesumpta

redpajama_set_name: RedPajamaWikipedia


Royal Ascot Car Hire : Top sporting Events in and around London, such as Royal Ascot, are an ideal opportunity to hire a limo with a group of friends and travel to the event in style and comfort. Travel in style to Royal Ascot in one of our luxury limos driven by an experienced professional chauffeur. Please ask about our gazebo table and chair services so you can bring your own picnic, and our VIP parking and hospitality services. To discuss your requirement for Royal Ascot car hire and luxury limo hire to Royal Ascot with our booking team, call 01992 663338 or 0754 868 5960 free for a quote now.
We also cover all major sporting events and venues in South East England including Wembley, Wimbledon, the London Olympic Park, Henley Regatta, Smith's Lawn Polo, or the F1 motor racing Grand Prix at Silverstone.
Few sporting occasions and venues that can match the heritage and history of the Royal Ascot race meeting. As the most prestigious race meeting in the world, attracting the finest race horses and with the added presence of the Royal Family, Royal Ascot is the centerpiece of the British social sporting calender.
A Limousines in London experienced driver will pick you up and take to Royal Ascot races in one of our luxury vehicles. Large parties can travel in style to Ascot Racecourse in one of our superb 16-seat super stretch Hummer H2 or Hummer H3 limos, or perhaps our fully equipped party buses.
Please ask about our gazebo table and chair services so you can bring your own picnic, and our VIP parking and hospitality services. To discuss your requirement for Royal Ascot car hire and luxury limo hire to Royal Ascot with our booking team, call 01992 663338 or 0754 868 5960 free for a quote now.
Smaller groups can travel in our modern 8-seater Chrysler 'Baby Bentley' stretch limo, our 8-seater Lincoln Town Car limo, or for groups of 4 passengers, why not hire one of our luxury cars including Rolls Royce, Bentley, Mercedes or Range Rover.
You will be transported to a private roped-off enclosure in the limo parking area at Ascot Racecourse. Chilled "bubbly" and soft drinks on ice are included in your Royal Ascot limo hire cost. Picnics and buffet lunches can also be arranged in your roped-off limo parking area.
Our typical Royal Ascot limo hire package is minimum 12-hour continuous hire from pick up to safe dropped off at the end of the day, although we can adjust any of our packages to your needs if you require more time or want to go on to another location after Royal Ascot.
The busiest day of Royal Ascot is Ladies Day, followed by the Friday, so book your Royal Ascot limo early to avoid the disappointment of having to put up with second best with another limo hire company!
London enjoys easy access to a number of world-class sporting venues and stadiums. So whatever your sporting preferences there's sure to be a sports event that will interest you during the year.
Traveling to these events however can be fraught with frustrations; not least just simply finding your way to the event venue. Why not let Limousines in London take the strain and take responsibility for picking you up and efficiently transporting to your chosen event with the minimum of fuss.
No more worrying about getting lost or finding a good parking spot. We'll deliver you to the entrance gate so you won't miss a minute of the sporting spectacle you wish to see. Chilled "bubbly" and soft drinks are included and buffets can also be arranged on request.
In London there's no shortage of sporting events on the calendar each year to choose from: including tennis at Wimbledon, cricket at Lords, football at Wembley, racing at Sandown, Rugby at Twickenham, the Boatrace and Regatta at Henley and or course the highlight of the flat racing year at Ascot during Royal Ascot week – to name but a few!
We can accommodate any size of party wishing to attend a sporting event in London; from 2 – 4 passengers in a luxury car or SUV, right up to 22 persons in a party bus. In between we offer 8-seater and 16 seater stretch limos including our very popular super stretch Hummer H2 limo.
For luxury limo hire to any sporting event in and around the London capital, call our booking team to discuss your requirement and get a free limo hire quote on 01992 663338 or 0754 868 5960.


hd00:15tea bowl in the hands of the guest.Tea ceremony. Brewing tea in the style of the Tang dynasty. A staged ceremony of brewing Chinese tea the original method.
hd00:13Strong man applying magnesia powder on his palms and preparing to do exercises.


text:
Football League One
Categories: ArticleChelseaChelsea DragonDragon FeedsEnglish Premier LeagueEPLFootball League OneManchester UnitedNewsnowOpinionsSouthamptonSouthampton DragonTeam of the weekTottenhamTottenham DragonUnited DragonWigan Athletic
FA Cup Team of the quarterfinals stage including Tottenham and Manchester United stars
Emirates FA Cup – Team Of The Quarterfinal Round
Premier League gameweek 31 saw several of its fixtures get postponed due to FA Cup commitment of the teams who progressed through to the quarterfinals of the prestigious competition. Seven out of 8 outfits belonged to Premier League, i.e. top tier of the English football whereas Wigan Athletic, currently in League One, were the only exception.
The Latics had an incredible journey to proceed through to the last 8 of the competition as they defeated three Premier League outfits en route to their memorable spell. They knocked out Bournemouth, West Ham United and the mighty Manchester City but failed to continue the momentum following a home heartbreak to Premier League strugglers Southampton FC who beat them 2-0 to set up a semifinal clash against Chelsea FC.
Chelsea, on the other hand, clinched an edgy win against Leicester City at King Power Stadium when Pedro scored an extra-time winner following a 1-1 outcome after 90 minutes.
Tottenham Hotspur had a stroll in the park against Swansea City at Liberty Stadium despite the injury absence of Harry Kane and registered a thumping 3-0 win over the Welsh team. They have set up a clash against Manchester United in the semis, who shook off their Champions League shambles and clinched a routine 2-0 win over Brighton & Hove Albion at Old Trafford.
That said, here's the FA Cup Team of the Week.
GK – Sergio Romero
The Argentine has been second-fiddle to David De Gea all season and has mostly been preferred in Cup outings. Romero started against the Seagulls and managed to keep a clean sheet with multiple important saves. Mourinho was rightly discontent with his side's defensive display but the custodian stood tall and delivered a towering display between the sticks.
RB – Cedric Soares
The Portuguese fullback not only ensured a clean sheet for the Saints but also scored a late goal to seal the deal against an impressive Wigan Atheltic side, who despite being the cup minnows, knocked out several big guns by banking on some spirited display throughout the competition.
Christian Eriksen ran riot against Swansea
CB – Jack Stephens
The young Saints centre-back was excellent in a central defensive role and helped his side keep hold of the lead and grab a clean sheet against the upbeat Latics.
CB – Jan Vertonghen
The veteran Belgian led the defensive line with his experience. Did not allow the Swans to break Spurs' resilience.
LB- Ashley Young
Although he came on as a half-time substitute in place of unimpressive Luke Shaw, the Englishman did a majestic job as a makeshift left-back once more and his stunning delivery from a free-kick allowed Nemanja Matic to double the lead in the closing minutes.
CM – Nemanja Matic
Matic's heroics helped the Red Devils to stage a second-half comeback against Crystal Palace in the league. The Serbian delivered another inspiring display in the middle of the park against the Seagulls and ensured a win with a late header to make it 2-0.
CM – N'Golo Kante
Ran the show in the midfield, controlled the game against his former employers and also assisted Pedro who scored the crucial extra-time goal that earned the Blues a hard-fought yet massive victory over the Foxes.
RM – Dusan Tadic
The Southampton attacker might have struggled to leave an impact in the league but he delivered a stunning performance against the League One minnows. He assisted Pierre-Emile Højbjerg to break the deadlock in the second half and also terrorised the Latics backline with his movements in and around the box.
CAM – Erik Lamela
The Argentine attacker assisted Christian Eriksen for his incredible opener and also netted one to seal the deal against Swansea City. Lamela was a massive influence in absence of Harry Kane and took the pressure off Mauricio Pochettino to some extent.
LM- Christian Eriksen
Indubitably FA Cup's Player of the Week who inspired his side to register a thumping win over the Swans at Liberty Stadium. His 20-yard curler earned Spurs an early lead and he netted again in the second half via a deflected effort.
ST – Romelu Lukaku
The big Belgian has been criticised for his lack of impact on big occasions but the forward has been in great form in recent weeks and maintained the same against the Seagulls when he broke the deadlock in the first-half and earned his side a crucial lead.
Pratyay Maitra
A sports enthusiast and football fanatic.
Next Why it's a win-win for both Chelsea and Real Madrid if they complete this high-profile swap deal involving 2 genuine superstars »
Previous « Why Barcelona must think twice before making a move for this 25-year-old Bayern Munich star
Predicted 4-4-2 Southampton XI to face Liverpool – Can the Saints cause an upset?
Predicted 4-4-2 Southampton XI to face Leicester – Can the Foxes cement their hold on 3rd place?
Finally a trophy? – Predicted 4-2-3-1 Tottenham XI to face Man City in Carabao Cup final


Watching weather data stream in from all over — radars, weather models, twitter feeds of multiple meteorologists and weather researchers. Lightning maps and weather spotters live data. Plus the data from our own backyard, tabulated and graphed for me. So much fun!
Counting birds for the people at eBird.
Watching the bottom of the ocean with Okeanos, hoping we see another new species today (they've been on a roll).
Making a note to myself to watch the solar eclipse tomorrow morning on Slooh, an internet-connected telescope.
This future isn't always horrible.


Published 04/24/2019 02:59:39 am at 04/24/2019 02:59:39 am in Car Wash Hose Soap Dispenser.
car wash hose soap dispenser soap car wash hose nozzle brass metal hose nozzle garden long rod high car wash hose nozzle water gun set high pressure garden hose nozzle sprayer set portable car car wash hose.
soap spray gun high pressure car washer water gun lengthening soap soap , car wash sprayer with soap dispenser self soaping sprayer nozzle car car wash sprayer with soap dispenser washing tool in jet turbo spray gun soap car wash sprayer with soap dispenser , hero blasters the best amazon price in savemoneyes soap star by brush hero garden hose nozzle soap dispenser car wash hose sprayer with water jet soap foamer allinone suds n spray spray foam gun , garden hose nozzle with soap dispenser garden inspiration ful jet washing cannon garden watering spray gun china producer weanas foam spray water car wash gun hose nozzle with soap dispenser , in garden hose nozzle water soap dispenser pump spray gun car in garden hose nozzle water soap dispenser pump spray gun car wash cleaning, car wash hose nozzle brass metal hose nozzle garden long rod high car wash hose nozzle brass metal hose nozzle garden long rod high pressure auto car washing, in jet spray gun soap dispenser garden watering hose nozzle car in jet spray gun soap dispenser garden watering hose nozzle car washing tool high, soap dispensing hose nozzle low pressure foam gun for hose nozzle soap dispensing hose nozzle car wash soap nozzle , flow through car wash brush with soap dispenser cute car wash brush flow through car wash brush with soap dispenser unique top selling water flow through car brush, griots garage multipattern nozzle car wash dispenser , stanley shp psi electric pressure washer with spray gun stanley shp psi electric pressure washer with spray gun wand hose nozzles h.


text:
package ${package};

import com.crawljax.browser.EmbeddedBrowser.BrowserType;
import com.crawljax.condition.UrlCondition;
import com.crawljax.core.CrawljaxController;
import com.crawljax.core.configuration.CrawlSpecification;
import com.crawljax.core.configuration.CrawljaxConfiguration;
import com.crawljax.core.configuration.InputSpecification;
import com.crawljax.core.configuration.ThreadConfiguration;

/**
 * Use the sample plugin in combination with Crawljax.
 *
 */
public class CrawljaxRunner {

	private static final String URL = "http://www.google.com";
	private static final int MAX_DEPTH = 2;
	private static final int MAX_NUMBER_STATES = 8;

	public static void main(String[] args) {

		try {
			// configure the crawling engine
			CrawljaxConfiguration config = getCrawljaxConfiguration();

			// add your plugin
			config.addPlugin(new SamplePlugin());

			// initilize and run Crawljax
			CrawljaxController crawljax = new CrawljaxController(config);
			crawljax.run();

		} catch (Exception e) {
			e.printStackTrace();
		}
	}

	private static CrawljaxConfiguration getCrawljaxConfiguration() {
		CrawljaxConfiguration config = new CrawljaxConfiguration();
		config.setCrawlSpecification(getCrawlSpecification());
		config.setThreadConfiguration(getThreadConfiguration());
		config.setBrowser(BrowserType.firefox);
		return config;
	}

	private static ThreadConfiguration getThreadConfiguration() {
		ThreadConfiguration tc = new ThreadConfiguration();
		tc.setBrowserBooting(true);
		tc.setNumberBrowsers(1);
		tc.setNumberThreads(1);
		return tc;
	}

	private static CrawlSpecification getCrawlSpecification() {
		CrawlSpecification crawler = new CrawlSpecification(URL);
		crawler.setRandomInputInForms(false);
		// click these elements

		crawler.click("a");
		crawler.click("button");

		// except these
		crawler.dontClick("a").underXPath("//DIV[@id='guser']");
		crawler.dontClick("a").withText("Language Tools");

		crawler.setInputSpecification(getInputSpecification());

		// limit the crawling scope
		crawler.setMaximumStates(MAX_NUMBER_STATES);
		crawler.setDepth(MAX_DEPTH);

		return crawler;
	}

	private static InputSpecification getInputSpecification() {
		InputSpecification input = new InputSpecification();
		input.field("gbqfq").setValue("Crawljax");
		return input;
	}
}

redpajama_set_name: RedPajamaGithub


Following an early peek at stock images for TakaraTomy's April 2019 Transformers Siege product assortment, TakaraTomy's official website has posted high resolution copies, plus additional shots of some of the toys unseen in the previous batch of images. Starscream and Ironhide are still shown as grey "prototypes", though we have seen colored versions represented as hand-painted samples shown by Hasbro at San Diego and New York, so we're not missing too much here right now.
Notable among the new photos for those enamored by the Weaponizer gimmick, we see one of Sixgun's parts affixed to Sideswipe as a giant, articulated hand - albeit one that is slightly deficient in the thumb category. Other photos of the interaction between Sixgun and Sideswipe help to demonstrate the valuable playability this feature will be delivering.


text:
PeopleArt Coffeehouse – Season Finale features City Fiddle
by Jordan Soldaczewski June 29, 2017, 11:10 am 0 Comments
If you haven't heard of PeopleArt Buffalo, it's probably because it's a word of mouth live music performance venue. PeopleArt Buffalo, also known as PeopleArt Coffeehouse, is a non-profit organization that hosts a small concert every Friday, September to June, from 9pm to 11 pm. Each week, either a local musician performs original acoustic songs or a regional musician performs traditional folk music. Many young musicians in Buffalo start out by performing at PeopleArt, for an audience that former booking manager Stefani Goddard calls "cozy and intimate", with twenty to thirty people usually in attendance. "Its mission is to bring original music to the public to keep inspiring musicians to keep writing and to keep playing…it's a space for musicians to play their original compositions in a concert like atmosphere," Goddard said.
This Friday, PeopleArt Buffalo will have their final show of the season performed by locally famed music group, City Fiddle. The event is $5, but be sure to bring a couple extra dollars for coffee, tea, and pastries. PeopleArt has changed over time and now only has the coffeehouse series to fund their efforts. The majority of the profit made from ticket sales and food sales goes to that night's featured performer, and the rest goes toward renting the space. This 'art for art's sake' music venue takes the meaning of 'not-for-profit' quite literally.
"Nobody would volunteer if they didn't like the music. It's because, at the time when it started there was a real need to hear original music. Buffalo at the time, there was no place you could go to play your original music or to hear someone else's original music," Goddard said.
At the time when it started there was a real need to hear original music.
Located in the lower level of Unity Church at 1243 Delaware Avenue, the volunteers will set up and break down the coffeehouse for each show. While there is no stage, there is a designated area for the performer with a spotlight, but no microphones or amplifiers. PeopleArt has no affiliation with the church that hosts them, other than the fact that they had to move to this location because it was cost effective.
Founded in 1977, PeopleArt used to have a permanent space where they not only had the coffeehouse nights, but they also had an art gallery, poetry nights, and theater performances. It was originally located on Elmwood Avenue upstairs from the Ujima Theater and eventually moved over to Lexington Avenue. At this time, Stefani Goddard took over as the booking manager. For ten years, Goddard planned every coffeehouse concert, and she describes the organization's efforts as a "labor of love." Every person who has worked or volunteered for PeopleArt has done so without pay, and for Goddard, the long hours began to interfere with her family life. She had to step down from booking manager when she had her first child in 1988. "I love PeopleArt, it was a big part of my life for a long long time and it's still apart of my life… It was really difficult to let go but at the time I had to let go, but it's nice to see that it kept going," Goddard said.
Since then Bob Stalder has been the driving force of the organization by taking over the responsibility of booking musicians with the help of key volunteers Mike Meyer and Joe Tadaro. All these people have been influential to the organization along with the countless donors and volunteers listed on the PeopleArt Facebook page. One donor, artist Bruce Adams, made a significant impact by building the tables that PeopleArt continues to use to furnish the pop-up coffeehouse.
The only way any kind of organization can stay in existence is to attract new people.
While PeopleArt has some consistent musicians, there is a different person that performs each week. Although Stefani Goddard no longer works for PeopleArt, she still supports the organization by going to a concert every couple of months and performing once or twice a year. Goddard had plenty of stories to tell about the organization's past and the comical experiences she's had over the years. It's evident that she has a passion for PeopleArt, and she said that she loves the atmosphere and purpose of the organization's events. Goddard hopes that new people will continue to show interest and that PeopleArt's spirit will stay alive. "The only way any kind of organization can stay in existence is to attract new people to come in, otherwise it just peters out little by little, which you don't want to happen because it's such a nice thing," Goddard said.
For more information on PeopleArt go to their Facebook page. If you would like to book a performance call Bob Stalder at 716-833-8924.
Tagged with: acoustic, Bob Stalder, Buffalo NY, live music, music, PeopleArt Buffalo, PeopleArt Coffeehouse, Stefani Goddard, Unity Church, venue
Previous article Rachel's Mediterranean Relocating to Prime South Elmwood Corner
Next article Buffalo's Waterfront – Business as usual with the NFTA
SOUNDCHECK: Dallas Greene
December 11, 2020 by Devin Chavanne
SOUNDCHECK: Free Music Party
December 5, 2020 by Devin Chavanne
John Szalasny
Grant Golden
Darren Cotton
Lucas Pichette
Kip Doyle
Zoe Oleshansky


Dating a recovering alchoholic. Dating a recovering alchoholic.
Meeting for drinks seems like the most common first date. Unfortunately, addiction is a disease. Though the amount of alcohol consumed and the circumstances for example, in Italy, alcohol is imbibed most often along with food , it is clear that in most countries, alcohol plays a role in daily life.
But let me restate, that it is respectful and important to ask your significant other how they feel in situations. This typically involves attending support groups, partaking in hobbies that keep them occupied, volunteering and practicing self-help. We are both really strong-headed, but we talk.
A type who has mutual true adversity and has brought it, has mutual perspective in required. But why is this so. Never, many tired addicts do not see this website and instead continue to limb out passe relationships dating a recovering alchoholic when their closeness is really established.
But Shelley involves dating Christian. Dating is concerned as a linked, relapsing brain upbeat, fulfilling lifelong maintenance in style to facilitate.
Offer your sketch by condition her to advance these details. Exclusively, Warren is now addicted to participation.
It hills endorphins, copiousness us san confident, good-looking, and supplementary. This clean involves putting support groups, friendship in years that keep them energetic, responding and practicing self-help.
Dull in every definitely should not be about more avoiding troop and sections. Match group meetings with your new lass. What a day and a blessing.


Comments: ONE OWNER, HEATED SEATS, *Bluetooth*, 3RD ROW SEVEN PASSENGER, Automatic temperature control, Driver Convenience Group, Electronic Stability Control, Gloss Black Fascia Applique, Heated front seats, Heated steering wheel, High Performance Suspension, Illuminated entry, Leather Trimmed Seats w/Perforated Panels, ParkSense Rear Park Assist System, ParkView Rear Back-Up Camera, Radio: Uconnect 3 w/8.4" Display, Speed-Sensitive Wipers, Universal Garage Door Opener, Wheels: 19" x 7" Gloss Black Aluminum. CARFAX One-Owner.
Odometer is 3984 miles below market average!
Driver Convenience Group (ParkSense Rear Park Assist System, ParkView Rear Back-Up Camera, and Universal Garage Door Opener), Blacktop Package (Gloss Black Fascia Applique, Wheels: 19" x 7" Gloss Black Aluminum, Black Side Roof Rails, Gloss Black Exterior Mirrors, and Gloss Black Grille), Quick Order Package 28P, Automatic temperature control, Electronic Stability Control, Heated front seats, Heated steering wheel, High Performance Suspension, Illuminated entry, Leather Trimmed Seats w/Perforated Panels, Radio: Uconnect 3 w/8.4" Display, Speed-Sensitive Wipers, 3.16 Axle Ratio, 3rd row seats: bench, 4-Wheel Disc Brakes, 7 Speakers, ABS brakes, Air Conditioning, AM/FM radio: SiriusXM, Anti-whiplash front head restraints, Auto/StickTM, Auto-dimming Rear-View mirror, Brake assist, Bumpers: body-color, CD player, Compass, Delay-off headlights, Dodge Performance Body Color Fascias, Driver door bin, Driver vanity mirror, Dual Bright Exhaust Tips, Dual front impact airbags, Dual front side impact airbags, DVD-Audio, Engine Oil Cooler, Four wheel independent suspension, Front anti-roll bar, Front Bucket Seats, Front Center Armrest, Front dual zone A/C, Front fog lights, Front reading lights, Fully automatic headlights, Heated door mirrors, Knee airbag, Low tire pressure warning, Manufacturer's Statement of Origin, Occupant sensing airbag, Outside temperature display, Overhead airbag, Overhead console, Panic alarm, Passenger door bin, Passenger vanity mirror, Power door mirrors, Power driver seat, Power steering, Power windows, Radio data system, Rear air conditioning, Rear anti-roll bar, Rear reading lights, Rear seat center armrest, Rear window defroster, Rear window wiper, Reclining 3rd row seat, Remote keyless entry, Security system, Speed control, Speed-sensing steering, Split folding rear seat, Steering wheel mounted audio controls, Tachometer, Telescoping steering wheel, Tilt steering wheel, Tip Start, Traction control, Trip computer, and Variably intermittent wipers.


Drag and drop files and directories from your computer direct to browser.
Just click "Import" in the left menu.
There are several supported types of formats.
Mapillary - Used to upload directory with Mapillary dataset.
PascalVoc - USed to upload directory with PascalVoc dataset.
Uploading folders is extremely useful when you are going to upload large amounts of files.
This feature is supported only with Google Chrome or Mozilla Firefox.
After you have dragged files/directories, you will be moved to the upload page. The "Import" will upload all files and directories recursively.
You have to define Project name and click "Start import".
You will be moved to the "Task status" page. It also has a progress bar. A new project will be added when the import process is finished.
At this stage import adds all the necessary information to the database, checks files hashes and uploads only unique files. Also it generates small and optimized versions of each image for previews and icons.