\section{Introduction}

\input{sections/01.Introduction}
\input{sections/02.Background.tex}
\input{sections/03.Deconstruction.tex}
\input{sections/05.SystemDescription.tex}
\input{sections/09.Discussion.tex}
\input{sections/10.Conclusion.tex}
\input{sections/11.Acknowledgements.tex}

\bibliographystyle{abbrv-doi}

\section{Introduction}

Visualization authoring tools give people varying degrees of control over how their data is visually represented even when they do not have enough time, resources, or skills to make custom visualizations programmatically. There are a growing number of these tools and, correspondingly, a growing number of approaches to designing them.

\noindent Popular tools, such as Excel~\cite{MSExcel} and Tableau Desktop~\cite{Tableau}, are primarily based on chart templates, automated mappings, and recommendation systems. These features are key for achieving a speedy authoring process and are beneficial for accessibility and ease of use. However, they can also impose barriers. Even for those with formal training in visualization, some types of automated approaches (such as templates) can be a hurdle to flexibility, get in the way of ideation processes, and may even take over the design lead. Recent research suggests that a constructive approach to visualization~\cite{Huron.2014a} has the potential to avoid some of these problems.\\

\noindent Constructive visualization promotes the idea of creating visual representations of data by assembling building blocks that are mapped to specific aspects of a dataset. This authoring strategy has been shown to empower people in their use of visualization without the need of formal training or other specialized skills (e.g., programming). Studies suggest that digital constructive visualization tools  can also promote a mindful design process in which people are encouraged to actively reflect on their design decisions~\cite{Mendez2017, Mendez2018}. \\

\noindent Despite these advantages, the authoring process of existing constructive tools---both tangible~\cite{huron2016, huron.2014b} and digital~\cite{Mendez2016}---does not scale well to larger datasets. Even for moderately simple visualizations, construction requires a significant amount of time, effort, and repetitive interactions~\cite{Mendez2017, Wun2016}. The problem is exacerbated as the amount of data to be represented (i.e., total records and attributes) grows.

\noindent Conventional visualization tools avoid the scalability issues of their constructive counterparts by automating certain steps of the visualization process. While automation may reduce the effort required from the designer, it tends to also interfere with personal agency. Automation can make the authoring process less incremental as the tasks delegated to the tool are often executed quickly and over large portions of data (e.g., entire attributes rather than individual values). This, in turn, reduces the transparency of the design process, as the tool's actions can be hard to follow, which can interfere with comprehension.\\

\noindent We join the growing body of work that investigates how to expand visualization authoring options. In particular, we present ReConstructor, a new point of exploration within the design space of digital constructive visualization tools. In ReConstructor, visualizations are constructed by instantiating their structural and functional components through the use of four interaction elements (\textit{objects}, \textit{modifiers}, \textit{activators}, and \textit{tools}). This design allows people create visualizations via a user-driven constructive approach that also eases the difficulty of working with larger datasets. That is, ReConstructor~supports a more scalable authoring process while keeping the agency of this process on the user's side.

\noindent More specifically, our work contributes: (1) a construction strategy for the design of scalable visualization tools based on four reusable interaction elements: objects, modifiers, activators and tools; (2) an explanation of how these interaction elements can be incorporated in the design of visualization authoring tools; and (3) the design and implementation of ReConstructor, a tool that supports the construction of visualizations through these elements.

\section{Related Work}
\label{section:relatedwork}

In this section, we discuss how the concept of construction has been used in the design of computer interfaces and, in particular, of recent visualization authoring tools.

\subsection{Constructive Theories}

Educational and learning theories such as Piaget's constructivism~\cite{Ackermann2001, Piaget2013}, Papert's constructionism ~\cite{Papert1991}, and Froebel's \textit{gifts}~\cite{stiny1980kindergarten} suggest that one way that humans discover the world is by manipulating simple objects and that we can construct knowledge and meaning from these experiences. These theories focus on personal experience, \quotes{\textit{where the learner is consciously engaged in constructing a public entity}}~\cite[p. 1]{Papert1991}, as the gateway to understanding and reflection.

\noindent Computational tools that implement constructive principles generally support processes based on an incremental, bottom-up strategy. This is related to the concept of \textit{emergence}, by which a complex entity arises as the result of interactions among smaller or simpler entities~\cite{abbott2004emergence}. Consequently, a constructive process is beneficial to show how a complex structure is built from the ground up, as the result of many small steps or sub-processes.\\

\noindent Constructive theories have been widely explored in environments that support the development of computational thinking skills such as Scratch~\cite{Dasgupta2015, maloney2010scratch, resnick2007all, Resnick} and Mindstorms~\cite{Papert1993} and, more recently, Google's Blockly library~\cite{Google2017}. In these tools, construction takes place with building blocks that animate interactive visuals.

\subsection{Construction in InfoVis}

Constructive Visualization~\cite{Huron.2014a}, a paradigm for visualization authoring grounded in Papert's, Piaget's and Froebel's theories, imports from them the idea of using physical \textit{tokens} (e.g., Lego blocks) that can be mapped to data and manipulated to compose tangible representations. iVoLVER~\cite{Mendez2016} implements constructive principles in a digital visual programming environment. Both Huron et al.'s tangible tokens (\cite{huron2016, huron.2014b}) and iVoLVER's marks support work with atomic data elements (e.g., individual values as opposed to entire data attributes). This leads to a bottom-up construction strategy in which the final design emerges as the result of several small-scale decisions and manipulations of the visualization elements.

\noindent Constructing visual representations of data from their atomic building blocks is in line with Bertin's semiotic views~\cite{Bertinb, Bertina}. To convey messages visually, Bertin's marks---graphical primitives such as points or lines---are configured in particular ways by mapping data attributes to their visual properties~\cite{Cleveland1984, Cleveland1987}.

\noindent ReConstructor's constructive approach also makes use of building blocks to provide access to a visualization's component (e.g., marks and visual properties) and to represent the operations that take place in different types of visual encodings (e.g., sorting, spacing).

\section{Enabling Construction}

For a visualization to be \textit{constructed}, the first step is to \textit{deconstruct} it into its modular components. We take this step to make use of the incremental nature of constructive visualization~\cite{Huron.2014a}.
To support the construction of a given visualization, the design of ReConstructor requires to identify the objects involved (e.g., visual marks, axes), their attributes (e.g., visual properties, labels), and their associated functionality (i.e., the processes associated to these objects such as \textit{sorting} or \textit{distributing}). We also have to pay attention to how these components interact. Having the visualization's objects, attributes, and functionality as modular components enables a constructive approach to visualization authoring.

\subsection{Interaction Elements}

ReConstructor's design is based on four fundamental interaction elements: objects, modifiers, activators, and tools. When combined, these elements enable reconstruction of a visualization from its modular components.\\

\noindent
An \textbf{object}
has a graphical representation, which supports visibility~\cite{norman:1988:psychology_everyday_things, norman:2010:gestural_usability} and allows for direct manipulation~\cite{Shneiderman1983}. They have various visible attributes (e.g., \prop{fill} and \prop{stroke} colors) and are by default inert---they do not cause any effect or interact with each other.\\

\noindent A \textbf{modifier} can take action on an object, such as applying the value of the object's attribute. It has the ability to change an object.
For example, a \prop{fill} modifier can change the fill color of an object and a \prop{stroke} modifier changes the color of the stroke that a pen tool produces. Modifiers can be of two types:
\vspace{-\parskip}
\begin{itemize}[leftmargin=0.6em, labelsep=0.15em]
	\setlength\itemsep{-0.5em}

	\item[-] \textit{Transient} modifiers change an object's attribute when placed on the object. Because the change in the object is visually conveyed, these modifiers are not graphically represented on the canvas. Transient modifiers include the \prop{stroke} and \prop{fill} color modifiers, as well as the \prop{shape} modifier that turns strokes into regular shapes.

	\item[-] \textit{Persistent} modifiers do not immediately change an attribute when placed on an object. Instead, they remain visible, attached to the object they modify. That is, they ``objectify''~\cite{Xia2016} attributes (e.g., \prop{width} or \prop{height}). Dropping a data dimension from a dataset onto the visual representation of a persistent modifier establishes a mapping between the corresponding data dimension and attribute. Persistent modifiers can add properties that are not inherent to strokes. For example, the \prop{label} modifier adds text to an object.\\
\end{itemize}

\noindent An \textbf{activator} carries a process that can be dynamically added to an object. That is, activators bring inert objects to life by turning them into \textit{tools}. For example, a \prop{push} activator carries the process of \prop{pushing} other objects around. Adding a \prop{push} activator to a squiggly stroke would give it the ability to push other objects when dragged. This activating strategy is relates to ActiveInk~\cite{Romat:2019:ActiveInk}, where ink annotations can be activated to operate on visual representations.\\

\noindent A \textbf{tool} is an object that has been activated. A tool can have one or more activators associated with it. For example, adding an \prop{ink} activator to a \prop{push}-activated stroke results in a pen that draws while pushing other objects out of the way.
Activators and persistent modifiers can also be removed from tools. For example, when a \prop{push} activator is removed from an object it no longer pushes other objects. Removing one activator does not affect the functionality added by other activators (if any).

\begin{figure}[b!]
	\begin{center}
		\includegraphics[width=\columnwidth]{./images/InkActivator.pdf}
		\caption{
			Constructing a \quotes{pen} tool from a user-drawn stroke (1) by attaching an \propCaption{ink} activator (2). The pen's (3, 4) ink is configured via \propCaption{color} modifiers (5--9).
		}
		\label{fig:constructingAPen}
	\end{center}
	\vspace{-2.5em}
\end{figure}

\begin{figure*}[b!]
	\begin{center}
		\includegraphics[width=\textwidth]{./images/TREES_ConstructingMarkAndBarChart.pdf}
		\caption{
			Constructing a bar chart using marks in the shape of trees.
			After drawing the outline of a tree (1), dropping the transient \propCaption{fill} color modifier (2) changes the fill color of the mark (3).
			Adding a \propCaption{label} (4) and a \propCaption{height} (5) modifier attach persistent representations of these modifiers underneath the mark (6), without any effect on the mark itself.
			Mapping the \propCaption{name} attribute to the \propCaption{label} modifier (7) adds the name of the first tree in the dataset at the top of the mark (8).
			Mapping the \propCaption{avg.height} attribute to the \propCaption{height} modifier (8) sets the height of the mark to the average height of the first tree in the dataset (9).
			Adding a \propCaption{replicate} activator to the tree and creates an interactive handle (10) that can be dragged right to create new visual mark that replicate the mappings of the original mark.
		}
		\label{fig:treechart-mark-bar}
	\end{center}
	\vspace{-2em}
\end{figure*}

\begin{figure}[t!]
	\begin{center}
		\includegraphics[width=\columnwidth]{./images/TREES_Dataset.pdf}
		\caption{
			A tabular dataset about trees, deconstructed into attributes (A) and records (B).
			This dataset can have a compressed representation (C), or an expanded representation showing its attributes (D) and records (E).
		}
		\label{fig:datasetVisualRepresentation}
	\end{center}
	\vspace{-2.75em}
\end{figure}

\subsection{An Illustrative Example}

To illustrate how we can use activators and modifiers to turn a passive object into an active tool, we describe the construction of a \quotes{pen} (Fig.~\ref{fig:constructingAPen}). In this case, we do not use any data. Instead, we simply illustrate the interaction aspects of our constructive approach through construction of a drawing tool. Here modifiers and activators have graphical representations that suggest their type and function and are added to objects via drag-and-drop gestures.\\

\noindent We first add to a user-drawn line (1) an \prop{ink} activator (2), which in this scenario carries a \prop{draw} function. The added activator appears as a directly manipulable graphical element that is attached to the activated stroke (3). The \prop{ink} activator of our example gives objects the ability to draw on the canvas when moved, converting the activated line into a \quotes{pen}. In this example, a pen generates scribbles on the canvas with the same width and color of the activated line (4). We then use a \prop{stroke} modifier (5) to change the activated line's stroke color to blue (6). Consequently, new movements of the pen produce blue scribbles (7). Subsequent modifications of the object's stroke color will also change the color of the pen's ink (e.g., to orange--8).
Dragging the brush icon out of our pen tool deactivates it (9). This step does not affect the line's current visual appearance or the scribbles previously drawn with it. Once deactivated, the stroke no longer draws when moved (10).

\begin{figure*}[t!]
	\begin{center}

		\includegraphics[width=\textwidth]{./images/TREES_DistributeAndSort_SINGLE_LINE.pdf}

		\caption{
			Distributing and sorting the marks of a chart.
			Drawing a horizontal stroke on top of the trees and adding a \propCaption{distribute} activator to that stroke (1) turns the stroke into a tool.
			This tool shows two handles (2) that can be dragged outward (3) to distribute and increase the horizontal space between marks.
			Adding a \prop{sort} activator (4) to the tool and mapping the \propCaption{avg.height} data attribute to it (5) sorts the marks based on their \propCaption{avg.height} value, in ascending order (6).
		}
		\label{fig:treechart-distribute-sort}
	\end{center}
	\vspace{-2em}
\end{figure*}

\section{ReConstructor}
\label{section:VisAssembler}

We now explain how objects, modifiers, activators and tools integrate in ReConstructor~to support visualization creation.

\subsection{The Interactive Environment}

In ReConstructor, visualizations are constructed within a canvas where people draw and move strokes via touch, pen or mouse. These strokes are the \textit{objects} of our domain. We use \textit{modifiers} to change objects' visual appearance. \textit{Activators} add functionality to objects, which enables the construction of tools to operate on other objects (e.g., a tool to spatially arrange a set of rectangles). As encouraged by the idea of Instrumental Interaction~\cite{BeaudouinLafon:2000wj}, ReConstructor~pays attention to visibility and enables direct manipulation~\cite{Shneiderman1983} of its interaction elements. Thus, both activators and modifiers are available as icons in a palette and are attached to (or removed from) objects via drag-and-drop---as in the illustrative example. When added to an object, activators are represented graphically, with an icon that suggests their function~\cite{norman:1988:psychology_everyday_things}. The same applies for persistent modifiers. Transient modifiers produce an immediate change on an object's visual appearance.

\subsection{Working with Data}

ReConstructor~currently supports tabular datasets.
We deconstruct tabular datasets into their structural components: data dimensions (i.e. columns) and records (i.e. rows).
A data dimension is a collection of all the values of a given column and is named after the column (Fig.~\ref{fig:datasetVisualRepresentation}.A).
A record is a set of attribute-value pairs (e.g., the record \prop{\{name: `Ponderosa Pine', avg.height: 478, avg.girth: 14\}} shown in Fig.~\ref{fig:datasetVisualRepresentation}.B) In ReConstructor, a dataset is represented on the canvas either in compressed (Fig.~\ref{fig:datasetVisualRepresentation}.C) or expanded form (Fig.~\ref{fig:datasetVisualRepresentation}.D).
When expanded, the dataset provides access to representations of its data dimensions (Fig.~\ref{fig:datasetVisualRepresentation}.D) and records (Fig.~\ref{fig:datasetVisualRepresentation}.E). These representations are draggable objects whose icon and color suggest the underlying data type: blue half-quotes for categorical attributes and green pound sign for quantitative ones. ReConstructor~organizes the dataset's components in an ordered sequence.
For example, the first value of the \prop{Name} dimension is \prop{Name[0]} and the first record of the dataset is \prop{Rows[0]}.
This order is relevant for some activators such as the \prop{tuple} and \prop{replicate} activators, as explained later.

\section{Constructing a Bar Chart}

We now use a running example of constructing a sorted bar chart made of tree shapes to show the use of our interaction elements.

\noindent A bar chart typically consists of two or more bars which:
\begin{enumerate*}
	\item have a height that represents an aspect of the data,
	\item are aligned to a common base,
	\item are optionally spaced for readability, and
	\item ideally, are labeled according to the data they present.
\end{enumerate*}\\

\noindent
\textbf{Step 1: Creating a visual mark with visual properties.}
Marks can be created from any object drawn on the canvas.
These objects have inherent attributes such as \prop{width}, \prop{height}, and \prop{stroke} and \prop{fill} colors.
Fig.~\ref{fig:treechart-mark-bar} (1--3) shows how to construct a visual mark from a hand-drawn stroke using the \prop{fill} transient modifier to set the fill color property of the tree to green. Other transient modifiers such as \prop{stroke} color and \prop{shape} beautification can be applied similarly.\\

\noindent
\textbf{Step 2: Mapping data dimensions to visual properties.}
We now show the label associated to the data record our tree mark represents, as well as map its height value to the mark's height attribute.
Fig.~\ref{fig:treechart-mark-bar} (4--9) shows the steps involved in establishing these data mappings.
We first add two persistent modifiers to the mark: \prop{label} (4) and \prop{height} (5).
Because no data is mapped to either modifier, there is no effect on the mark itself but icons representing these modifiers are attached underneath the mark (5, 6). These icons can be used to map different data dimensions to visual properties.
To establish a data mapping, we drag a data dimension from the dataset and drop it onto the icon of a persistent modifier. When mapping the \prop{Name} attribute to the \prop{label} modifier of the mark (7), the name of the first record in the dataset (\prop{Name[0]}) is attached to the tree (8).
We then map the \prop{Avg.Height} data dimension to the \prop{height} modifier (8). This changes the height of the mark using the value of the first record in the dataset (\prop{Avg.Height[0]}).\\

\noindent
\textbf{Step 3: Replicating the defined mappings.}
At this stage, we have created a single mark that represents the first record of our dataset.
Fig.~\ref{fig:treechart-mark-bar} (9--11) shows how we use a \prop{replicate} activator to avoid having to manually define mappings for each record of the dataset.
We first add the \prop{replicate} activator to the mark (9). This adds a handler to the right of the mark's visual properties (10, yellow encircled arrow). This step turns the newly activated object into an interactive tool that can replicate itself.
A numeric value on top of the yellow handler indicates the number of additional data records that can be replicated.
We then drag the handler from left to right to create new marks (11). The more we drag, the more visual marks are created.
Each newly created mark replicates the mappings of the original one using subsequent data records (e.g., the first replicated mark uses the values \prop{Name[1]} and \prop{Avg.Height[1]}).
As new marks are replicated the number of available records to replicate shown on top of the handler decreases.
When this value is zero the handler is grayed out and cannot be dragged  further.\\

\noindent
\textbf{Step 4: Distributing and sorting the trees.}
\label{section:distributingAndSorting}
Now that we have constructed a bar chart, we want to rearrange the trees to evenly add some space between them, and to sort them in ascending order.
We achieve these actions by creating another interactive tool, as shown in Fig.~\ref{fig:treechart-distribute-sort}.
We draw a horizontal stroke on top of the trees and add a \prop{distribute} activator to it (1).
Upon adding the \prop{distribute} activator, two interactive handles appear at the start and end points of the space spanned by the marks intersected by the stroke (2).
Activating the stroke turns it into a distributing \textit{tool}. Moving its handles (3) increases (or decreases) the horizontal space between the trees.
The last step is to sort the trees in ascending order.
We add a \prop{sort} activator (4) to the horizontal stroke and drag the \prop{Avg.Height} data dimension of the dataset onto it (5).
This sorts the trees in ascending order by default (6). Tapping the \prop{sort} activator's icon changes the sorting direction (to descending) if needed.

\section{Discussion}

We chose to explore a variation of a digital constructive approach to creating visualizations. A constructive approach requires components from which the visualization can be built. Previous constructive approaches~\cite{Huron.2014a, Mendez2016} have decomposed the data into individual data entities which can be represented as tokens. Construction can then proceed with these tokens. While this has shown to empower people and promote data and visualization understanding~\cite{huron.2014b, Mendez2017, Wun2016}, people also object to the tedium of moving individual tokens~\cite{Wun2016}.\\

\noindent In ReConstructor~we consider leveraging the tabular data structure in our deconstruction. This approach to deconstruction, allows the user to make use of this structure through our replicator activator to  pull related data entities unto the canvas. This helps alleviate some of tedium of placing the single data items, and points to a possible direction for combining the advantages of constructive visualization with a more scalable interactive approach.

\subsection{Flexibility and Sequencing}

One of the key points of construction is the potential to achieve similar outcomes by combining atomic elements in multiple ways. In contrast to tangible tools, this is particularly true for digital construction, as software-supported processes can be notoriously mutable. Our ongoing observations show that people expect a more relaxed workflow regarding the order of operations they executed in ReConstructor. This is in line with studies that have shown that, when the freedom is available,
humans do not follow particular sequences during design but rather
opt for personal variations that change with the problem at hand [19]. Developing ReConstructor, however, made evident that full flexibility can be hard to achieve in a software solution. It appears that some order is required to achieve a construction. In ReConstructor we make use of the order used to store the tabular data.

\subsection{Granularity, Scalability and Agency}

ReConstructor~operates on data dimensions—rather than individual values like other constructive tools. In combination with our replication strategy, working at this coarser granularity level provides some scalability. We also circumvent scalability issues by enabling people to create tools to operate on the tokens they use. One can construct, for example, a \quotes{sorting tool} to sort the marks of a visualization according to a specific data dimension. This is a unique aspect of ReConstructor that places more responsibility (hence, agency) in the hands of the designer. It also differs from most conventional visualization tools in that although certain operations can be speeded up, they are not fully delegated to the tool but rather constructed by the designer.

\section{Conclusion}

ReConstructor~provides a new variation on constructive visualization that supports the creation of visualizations based on four reusable interaction elements---objects, modifiers, activators and tools. In creating a visualization with ReConstructor~visual marks are constructed by attaching modifiers to hand-drawn canvas \textit{objects}. Persistent \textit{modifiers} provide mechanisms for establishing data bindings, achieving data-driven modifications of an object's visual appearance. \textit{Activators} can also be added to the visual marks to enable the construction of \textit{tools} that can operate on the visualization's components. In combination, these four interaction elements enable the creation of visualizations through a constructive authoring process.


\subsection{Error Recovery}
\label{sec:error-recovery}
There are several types of potential errors in executing a query:

\noindent (1) The query cannot be accepted by the grammar.
For example, out-of-context input (\query{What time is it now}) and unsupported functionality (\query{Split the data into two halves}) would receive grammar rejection;

\noindent (2) The query is grammatically correct but invalid based on the dataflow context, possibly due to incorrect references of dataset and diagram elements.
For example, the user may attempt to show data from a non-existing node, \emph{e.g.}\xspace asking to \query{Highlight the selected cars from the scatterplot} when there is no scatterplot in the dataflow.
Such errors are captured at the query pattern completion step.

\noindent (3) The query is executed fully but does not meet the user's expectation.
For example, \query{Show the data} by default creates a scatterplot but the user instead wants a heatmap, or \query{Merge these two nodes} merges an unexpected pair of nodes when ``these'' appears to be a vague reference (the system chooses two nodes with the highest focus scores).

Upon the first two types of errors the system displays a message and asks for a query correction.
For the last type of error it is up to the user to adjust the dataflow diagram.
Since the user is simultaneously using the underlying VisFlow\xspace DFVS while using FlowSense\xspace,
she always has the flexibility to undo the FlowSense\xspace action or to make partial adjustments when the NLI does not yield exactly the desired outcome.

\subsection{Function Classification}
\label{sec:function-classification}
FlowSense\xspace uses keyword classification to identify the semantic meaning of words in the NL query and uses this information to decide a proper VisFlow\xspace function to execute.
For instance, the verb ``show'' is a synonym of ``visualize'', ``draw'', etc.
These words indicate the intention to create a visualization.
Meanwhile, ``find'' may implicitly specify a data filtering requirement and is similar to ``filter''.
We compute the Wu-Palmer similarity scores~\cite{WuZ94} between words and use the measured scores to classify words in the NL query that have close meaning to a set of pre-determined VisFlow\xspace function indicators.
The implementation of the similarity scores is based on WordNet~\cite{FellbaumC98} and NLTK~\cite{NLTK}.
\subsection{Special Utterance and POS Tagging}
\label{sec:tagging}
Special utterances have remarkable roles in executing a VisFlow\xspace function.
Their tagging is performed on the fly when the user types the query.
For typo tolerance,
FlowSense\xspace employs approximate matching and checks each $k$-gram in the query (where $k$ may range from $1$ to the maximum special utterance word length) against all special utterances using case-insensitive Levenshtein distance~\cite{LevenshteinVI66,NavarroG01}.
We divide the distance over the string length and use the ratio to mitigate the fact that longer strings are more prone to typos.
We find a $k$ value of $2$ or $3$ and a ratio threshold of $0.2$ work well in practice.

\input{sections/query-execution/fig-exec}

In addition to recognizing special utterances, FlowSense\xspace also performs POS tagging on the query with CoreNLP.
Each token receives a POS tag as shown in \autoref{fig:pattern}.
POS tags are used to generalize the FlowSense\xspace grammar.
For example, many prepositions can be used interchangeably, \emph{e.g.}\xspace ``selection {\it of} the plot'' is equivalent to ``selection {\it from} the plot''.
Instead of having one rule for every preposition, the grammar uses a generic variable that matches any preposition.
POS tagging helps analyze the basic semantic structure of a query.

\subsection{Diagram Update}
Once a query is successfully completed, FlowSense\xspace performs the VisFlow\xspace function(s) with the given function options.
This typically results in the creation of one or more nodes, \emph{e.g.}\xspace the visualization function creates one plot while the highlighting function creates three nodes (Table~\ref{tab:functions}).
FlowSense\xspace may also update existing nodes without creating any new nodes, \emph{e.g.}\xspace when the user only changes rendering colors.
Additionally, a query may operate on multiple existing nodes at once, \emph{e.g.}\xspace linking and merging two tables create edges between two nodes.
Operating on multiple nodes together helps simplify dataflow interaction, as these operations previously require multiple drag-and-drop interactions.

After new nodes and edges are created, the diagram may become more cluttered.
FlowSense\xspace locally adjusts the diagram layout after each diagram update.
We use a force-directed layout modified from the D3 library~\cite{D3} that manipulates the vicinity of the current diagram editing focus.
We extend the force to take rectangular node sizes into account so that larger nodes such as embedded visualizations have stronger repulsive force for avoiding node overlap.
User-adjusted node positions are remembered by the system,
and the layout algorithm avoids moving nodes that have been positioned by the user.
Currently FlowSense\xspace does not look for an optimal dataflow layout.
We leave layout improvement~\cite{BatiniC86} for future work.

\subsection{Query Pattern Completion}
\label{sec:pattern-completion}
After the parser identifies the existing key components of a query,
FlowSense\xspace attempts to fill in the blanks where information is missing using default values or the diagram editing focus.

\subsubsection{Finding Default Values}
Query components may be completed using default values.
Function options may have defaults.
For instance, FlowSense\xspace automatically chooses two numerical columns to visualize in a scatterplot triggered by a simple query \query{Show a scatterplot}.
Note that within a DFVS decisions like this can easily be changed by the user.
So FlowSense\xspace does not necessarily need to make a best guess.
Similar decisions include completing port specification.
By default FlowSense\xspace applies the newly created filter to all the data a visualization node receives, rather than the data subset interactively selected in the visualization.
Sometimes the default values may even be empty.
A query like \query{Filter by mpg} results in FlowSense\xspace creating a range filter on the mpg column with no filtering range given (\emph{i.e.}\xspace a no-op filter placeholder).
The user can then follow up and fill in the filtering range via the DFVS interface.

\subsubsection{Finding Diagram Editing Focus}
Whenever the user expands the dataflow diagram there always exists an editing focus, though often the focus is implicit.
For example, when the query contains a phrase like ``from MyChart'', the focus (\emph{i.e.}\xspace the source node of the query) is explicitly given.
However, users tend to neglect the source or target nodes in their queries, especially when there is a sequence of commands that together complete a task.
When a query does not have explicit focus,
FlowSense\xspace derives the user's implicit focus based on user interaction heuristics.
We compute a focus score for every node $X$ by:
\begin{equation}\nonumber
{\textit score}(X) = {\textit activeness}(X,t) + \alpha(1 -
{
\frac{1}{1 + e^{-({\textit distanceToMouse}(X)/\gamma - \beta)}}
}).
\end{equation}

The activeness of $X$ is re-iterated upon every user click in the system:
\begin{align}\nonumber
{\textit activeness}(X,t) = {\textit activeness}(X,t-1) / 2 + {\textit click}(X, t),
\end{align}
where $\text{\it click}(X,t) = 1$ if the $t$-th click is on $X$ and $0$ otherwise.
This definition measures how actively a user focuses on a node by how many times she recently clicks on it, as well as how close it is to the mouse cursor.
The activeness derived from user clicks decreases exponentially over time, while the closeness to mouse dominates under a small distance with a shifted sigmoid function\footnote{See the appendix for more explanation on the characteristics of the diagram editing focus heuristics.}.
We find the parameters $\alpha = 2, \beta = 5, \gamma = 500$ achieve good result.
FlowSense\xspace chooses the node with the highest focus score to be the diagram editing focus.
If multiple source nodes are required (\emph{e.g.}\xspace in a merge query), FlowSense\xspace selects the nodes in the order of their decreasing focus scores.

The focus may also be required by node type references.
For instance, the user may input \query{show the data from the scatterplot}, in which ``scatterplot'' is a reference by node type that describes a scatterplot node existing in the dataflow diagram.
In case of a tie during the node type search, \emph{e.g.}\xspace there are multiple scatterplots in the diagram, the nodes with higher focus scores are chosen.

\subsubsection{Query Completion Ambiguity}
There may be multiple syntactically correct ways to execute a same query.
Consider the query \query{Show the cars with mpg greater than 15} applied on a visualization node.
From the grammar perspective the parsed outcome has no ambiguity: Apply an attribute filter and visualize the result.
However, there are two ways of execution:
One is to create a filter and then visualize the filtered cars in a new visualization;
Alternatively we may apply the filter on the input of the current visualization so that the existing visualization shows only the filtered cars.
Both can be desired under some circumstances.
FlowSense\xspace has the default behavior that prefers filtering the input when the source node is a visualization,
which we find empirically more intuitive.
Such ambiguity can often be resolved with a slightly refined query, \emph{e.g.}\xspace \query{Show the cars with mpg greater than 15 from the plot},
which would explicitly indicate that the filter should be applied to the output of the existing visualization.

\section{Query Execution}
FlowSense\xspace is built as an extension to VisFlow\xspace.
The user may activate the NLI at any time while working with the DFVS.
The user may either type the query in the input box or use the speech mode that is implemented with HTML5 web speech API.
In this section we introduce the query execution workflow as depicted by \autoref{fig:exec}.

\input{sections/query-execution/tagging}

\input{sections/query-execution/function-classification}

\input{sections/evaluation/fig-visionzeroa}

\input{sections/query-execution/pattern-completion}
\input{sections/query-execution/diagram-update}
\input{sections/query-execution/error-recovery}

\section{Discussion and Limitations}
\label{sec:discussion}

\subsection{Scalability and Generalizability}
Technically there are many ways to create a set of rules that implement the same dataflow function.
Following the active development and enhancement of the system functionality, from time to time grammar rules can be combined and rewritten to make the grammar more concise.
We keep iterating and refining the FlowSense\xspace grammar to expand its functionality.
FlowSense\xspace currently includes about 200 variables and a rule set of around 500 rules in its grammar.
Our grammar development practice employs continuous integration and maintains a test set (currently of 131 test queries) to ensure that all categories of VisFlow\xspace functions may execute properly during iterations and extensions of the grammar.
Approximately 10 to 20 rules need to be added to support a new dataflow function category.

Though the grammar rules of FlowSense\xspace are coupled with the underlying VisFlow\xspace functionality,
its approach of utilizing special utterance placeholders is generalizable to other dataflow systems that employ similar modular component design.
Once the data- and diagram-independent dataflow elements are identified,
these elements can be represented by special utterances in the grammar and dataflow implementation can subsequently be extended to process them.
For example, we may extend the grammar to support more data processing power obtainable from a computational dataflow system like KNIME~\cite{Knime}.

\subsection{User Behavior and Engagement}
The effectiveness of a grammar-based semantic parser couples with the grammar design.
One design flaw in the grammar may result in unexpected rejections of seemingly acceptable queries.
Despite careful grammar design, the user is likely to come up with questions that exceed the scope of the grammar.
However, we find that users are willing and able to refine rejected queries with a small number of trial-and-error attempts.
Besides users may become more proficient with the NLI after reading query examples so as to understand the NLI capability.
Yet showing too many examples may limit the user's thoughts and forfeit the benefit of using an NLI.
We would like to further study user behavior regarding NLI usage in DFVS in the future to better identify when and what query examples need to be provided.

We also observe that users tend to perform composite queries and ask for batch operations using the NLI.
With traditional mouse/keyboard interaction, the results of such queries have to be achieved by a sequence of interactions.
FlowSense\xspace increases the data exploration efficiency by naturally enabling batch operations.
In fact, we notice some users were able to repeat successful short queries that achieved the most batched result.
The convenience of using NL to carry out multiple operations may improve the user's engagement~\cite{Cohen92},
provide interaction ``shortcuts'',
and make dataflow features more accessible by simplifying the creation of rather complicated sub-diagrams, \emph{e.g.}\xspace ``highlighting''.

\subsection{Technique and Scope}
We prefer semantic parsing to deep learning mainly because the latter has a bottleneck of requiring a large volume of training examples.
Though there are benchmark datasets for general NLP, there has not yet been a training set catered for visualization-oriented NLI or DFVS.
In the future with more users working with the NLI, we may collect more user queries that constitute a rich training set in order to support methods like neural networks for text classification~\cite{YoungT18}.

Currently FlowSense\xspace only works with dataflow diagram editing.
But it may be desirable for the NLI to answer analytical questions such as \query{Does the vehicle speed decrease over years in NYC?} by creating a visualization like \autoref{fig:visionzero-a}(4).
To that end we need further research on the dataflow functions and their application to answering analytical questions.
One possible direction is to study how DFVS diagrams can be constructed for knowledge-based Q\&A~\cite{ClarkP99}.

\subsection{Grammar}
FlowSense\xspace applies a semantic parser to map an NL query to one of the VisFlow\xspace functions based on an elaborate grammar designed for these functions.
The grammar is context-free~\cite{SipserM12} and formally defined as a 4-tuple $G = (V, \Sigma, R, S)$.
$V$ is a finite set of variables.
$\Sigma$ is a finite set of terminals.
A terminal represents an English word or phrase.
$R$ is the rule set that defines how a single variable matches an ordered list of terminals and variables (possibly itself in a recursive rule).
Below is an example rule:
\begin{align}\nonumber
\var{Visualization} \rightarrow \var{ShowVerb}~\var{Columns}~in~\var{VisualizationType}
\end{align}
In this rule, $\var{Visualization}$ is a high-level variable that matches a query that requests a visualization.
$\var{ShowVerb}$ matches a verb that has a meaning similar to ``show''.
$\var{Columns}$ matches one or more columns from the data.
$\var{VisualizationType}$ stands for a phrase that describes a visualization metaphor such as scatterplot or parallel coordinates.
The token ``in'' is a terminal symbol that comes from the NL input directly.
The example rule above is simplified for the convenience of explanation.
In practice, a rule often matches against generic variables rather than a specific word.
$S$ is the start variable that expands to other variables to match the whole query.

The grammar of the FlowSense\xspace semantic parser attempts to derive an input query by recursively searching for all possible matches (up to a preset limit) of the grammar rules.
This procedure is called {\it derivation}~\cite{BerantJ13}.
FlowSense\xspace uses the semantic parsing implementation from SEMPRE\xspace.
It also uses the Stanford CoreNLP~\cite{ManningCD14} toolkit that is built into SEMPRE\xspace for special utterance tagging.
The variables and rules (\emph{i.e.}\xspace SEMPRE\xspace formulas) are defined in SEMPRE\xspace grammar files.

\subsubsection{Special Utterance Placeholders}
The FlowSense\xspace grammar consists of static grammar rules and the special utterance placeholders.
The special utterance placeholders are at runtime dynamically replaced by their corresponding dataflow elements.
Therefore, the FlowSense\xspace semantic parsing is independent of the dataset, the dataflow diagram, and the analytical tasks.
The rules are generalizable across domains:
No new rules need to be created when the system switches to new datasets or tasks.

For example, FlowSense\xspace uses the generic variable $\var{column}$ in its grammar as a special utterance placeholder.
At runtime, a real column name (\emph{e.g.}\xspace ``mpg'') is automatically extracted from the dataset.
FlowSense\xspace identifies column names on the fly as the user types the query.
``mpg'' would show up as a tagged column, and then matched with $\var{column}$ by the parser.
A reverse mapping is performed from the placeholder to the particular column after query parsing so that the system may operate on that column.

Using special utterances in the grammar has several benefits.
First, special utterances enable VisFlow\xspace functions to operate on elements that are important for dataflow diagram editing and visual data exploration.
Second, it makes the grammar set small as rules may be written with generic variables rather than specific dataset or diagram content.
Last but not least, the real-time tagging of special utterances provides important feedback to the user about what operations are available in the system and how the NLI interprets the query.

\subsubsection{Derivation Ambiguity}
\label{sec:derivation-ambiguity}
It is possible to have ambiguity when multiple possible query derivations exist, which can be defined as syntactic ambiguity~\cite{TongG15}.
For example, FlowSense\xspace uses wildcard variables to match general {\it table row} references.
Over the Auto MPG dataset, the token ``cars'' from \query{Show a plot of cars} describes the user's understanding of data entities but should be only treated as table rows from the NLI perspective.
Meanwhile, the token ``horsepower'' from \query{Show a plot of horsepower} is a special utterance and should be treated as a column to visualize.
Therefore a wildcard rule that matches ``cars'' as table rows may also match ``horsepower'', resulting in the second query getting improperly executed.
We could handle this case by creating a wildcard variable that rejects a special utterance token.
Nevertheless, such a design would lead to a larger number of variables and rules in the grammar, which are harder to maintain and develop.
Therefore we choose to resolve certain syntactic ambiguity in the parsing phase with supervised learning on a weight vector $\mathbf{w}\in \mathbb{R}^d$ that gives the probability of derivations based on input utterances.
Stochastic gradient descent (SGD) is employed to optimize the multiclass hinge loss objective~\cite{TaskarB03}, as introduced by Liang et al.~\cite{LiangP14} in the SEMPRE\xspace framework.
The objective is given by:

\begin{align*}
\min_{\mathbf{w}} \sum_{(x,y)}{\max_{y'}{\{\mathbf{w}\cdot{\textit feature}(x, y') + {\textit penalty}(y, y')\}} - \mathbf{w}\cdot{\textit feature}(x, y)}
\end{align*}

In the above, $x$ is the input query, $y$ is the preferred derivation, and $y'$ is a derivation choice.
The pair $(x, y)$ is iterated over all training data.
The feature of a derivation, {\it feature}$(x, y)$, maps the pair $(x, y)$ to a $d$-dimensional space and is determined by the applied rules in the derivation.
{\it penalty}$(y, y')$ is $0$ if $y = y'$ and $1$ otherwise.
The objective function has a penalty for possible choices of incorrect predictions that are within a margin of one from the correct predictions.
The parser fits the training examples by giving intended derivations higher probability so that they are preferred in case of ambiguity.
In particular, the rule that expands to a column special utterance will be preferred over a rule that expands to a wildcard.
Note that we only apply this training to facilitate the simplicity of the FlowSense\xspace grammar and reduce the number of required rules.
The training cannot address the ambiguity in natural language itself at large.
We were able to use a small training set of fewer than twenty examples to guide the preferred derivation in case of syntactic ambiguity for a rule set of around 500 rules.
This is feasible because the FlowSense\xspace rules are independent of data and dataflow diagrams.
The training set only needs to guide the semantic parser to focus on certain important grammatical features, such as special utterances or word proximity.

\subsection{Auto-Completion}
\label{sec:auto-completion}

The usability of an NLI is closely related to its discoverability.
It is desirable that when the query is partially completed,
the system is able to provide hints or suggestions to the user about valid queries that include the partial input.
This has been a requested feature in prior NLI user studies~\cite{TongG15}.
We therefore develop an auto-completion algorithm in FlowSense\xspace to enhance its usability and discoverability.
When the user types a partial query and pauses, the system triggers query auto-completion automatically.
The auto-completion may also be invoked manually with a button press.
\autoref{fig:flowsense-ui}(a) shows the auto-completion suggestions in the FlowSense\xspace input box.

Auto-completion has been implemented in other visualization NLI, such as Eviza~\cite{SetlurV16}.
Eviza applies a template-based auto-completion, in which the system attempts to align user input to available templates.
Here we take a similar approach by creating a set of query templates with around one hundred queries.
Upon an auto-completion request, the algorithm searches through all possible textual matches between the user's partial query and a prefix of some template.
All matched queries are then sent to the FlowSense\xspace parser for evaluation.
If a query is accepted, it becomes an auto-completion candidate.
Some of the queries contain value placeholders and the user is expected to fill in those values ([string], [number] in \autoref{fig:flowsense-ui}(a)).

We also design a token completion algorithm that matches the partially typed word against available special utterances.
This helps speed up query typing with respect to the dataflow context.
The user may use the tab and arrow keys to select token completion candidates as in a programming IDE.
For example, when ``scatter'' is typed it can be completed to the available visualization type ``scatterplot'' (\autoref{fig:flowsense-ui}(b)).
Token auto-completion reduces typing workload and helps remind the user of the DFVS capability and the current dataflow diagram elements.

\subsection{VisFlow\xspace Functions}
\label{sec:functions}
To create an NLI for VisFlow\xspace, we first studied a sample diagram set that includes 60 dataflow diagrams created by 16 VisFlow\xspace users from their recorded VisFlow\xspace sessions.
These diagrams cover a wide range of VisFlow\xspace usage scenarios and deal with various types of datasets.
We identify a set of frequently appearing sub-diagrams and categorize them into six major categories as listed in \autoref{tab:functions}.
The construction of these sub-diagrams are defined as the {\itVisFlow\xspace functions}.
By implementing the VisFlow\xspace functions, FlowSense\xspace essentially supports the building blocks of visual data exploration in VisFlow\xspace so that analyses rendered by VisFlow\xspace native interactions can be carried out with FlowSense\xspace.
These functions also reflect the fundamental analytical activity defined in information visualization task taxonomies~\cite{ShneidermanB96,AmarR05}.
\autoref{tab:functions} explains the usage of each VisFlow\xspace function and shows several sample queries.

In addition to the six major categories, FlowSense\xspace also supports many utility functions such as adding/removing dataflow nodes/edges, undo/redo, loading datasets, etc.
Though these functions also enhance the usability of the system, we omit them here as they are indirectly related to visual data analysis.

\subsection{Dataflow Context and Special Utterances}
\label{sec:dataflow-context}
It is important to make the semantic parser aware of the dataflow context, such as the dataset loaded and the nodes in the dataflow diagram.
FlowSense\xspace extracts a special group of tokens called the {\it special utterances} from NL input.
Special utterances are words that refer to entities in the dataset or the dataflow diagram.
They are the arguments and operands of VisFlow\xspace functions.
FlowSense\xspace recognizes table column names, node labels, node types, and dataset names as special utterances.
For the query shown in \autoref{fig:pattern}, FlowSense\xspace identifies ``mpg'', ``horsepower'', and ``origin'' as table columns, ``MyChart'' as a node label, and ``parallel coordinates'' as a node type.
The special utterances identified by FlowSense\xspace are shown in colored tags in the FlowSense\xspace input box (\autoref{fig:flowsense-ui}).
Each distinct color represents one special utterance type: green for table column, light green for node label, purple for node type, and light blue for dataset name.
The colors are applied consistently throughout the user interface.

\section{Semantic Parser}
\label{sec:design}
In this section, we define the building blocks of the semantic parser:
the VisFlow\xspace functions that can be specified by NL,
the definition of the parsing grammar,
and the general query pattern the parsing algorithm expects.
For concept illustration we use the Auto MPG dataset\footnote{http://archive.ics.uci.edu/ml/datasets/Auto+MPG} throughout the paper,
which has information about cars in 9 columns, including mpg, horsepower, origin, etc.

\input{sections/semantic-parser/functions}
\input{sections/semantic-parser/special-utterance}
\input{sections/semantic-parser/grammar}
\input{sections/semantic-parser/query-pattern}

\input{sections/semantic-parser/fig-flowsense-ui}

\input{sections/semantic-parser/auto-completion}

\subsection{Query Pattern}
The main goal of FlowSense\xspace is to support progressive construction of dataflow diagrams.
We studied the creation process of the VisFlow\xspace diagrams in our sample diagram set and empirically identified a common pattern with five key {\itshape query components} that all VisFlow\xspace functions may contain:
{\itshape function type}, {\itshape function options}, {\itshape source node(s)}, {\itshape target node(s)}, and {\itshape port specification}.
This pattern is illustrated in \autoref{fig:pattern} with a sample query \query{Visualize mpg, horsepower, and origin of the selected cars from MyChart in a parallel coordinates plot}.
In this query, the verb ``visualize'' implies applying a visualization function.
The three columns ``mpg, horsepower, and origin'' indicate the options (\emph{i.e.}\xspace what to visualize) for the visualization function.
The phrase ``from MyChart'' tells the system the location of the data to be plotted and provides source node information.
The phrase ``in a parallel coordinates plot'' indicates a new visualization node of the given visualization type is to be created as the target node.
As VisFlow\xspace explicitly exports interactive data selection from visualization nodes, the phrase ``selected cars'' is a port specification that further describes that the user wants to visualize the selection from MyChart and the new visualization node should be connected to the selection output port of MyChart.

The grammar of FlowSense\xspace includes a variable hierarchy that matches the five key components of an NL query.
\autoref{fig:pattern} illustrates the parse tree that derives the sample query.
The variables involved in the derivation are shown in the parse tree, in which rule expansions are bottom-up.
A variable may carry information for multiple query components.
We design a broad set of variables and rules that are able to not only accept queries with a particular component order, but also their different arrangements.
For instance, ``Show mpg and horsepower in a scatterplot'' is equivalent to ``Show a scatterplot of mpg and horsepower''.
They both can be accepted by FlowSense\xspace.
FlowSense\xspace is also able to derive multiple functions from one single query and execute their combination, \emph{e.g.}\xspace \query{Show the cars with mpg greater than 15 in a scatterplot} infers both visualization and filtering functions.

A query may not necessarily contain all the five components explicitly.
For example, the user may simply say \query{Show mpg and horsepower} without mentioning any source node or target visualization type.
FlowSense\xspace may automatically locate source and target nodes in its query pattern completion phase (\autoref{sec:pattern-completion}).
An NL query may also contain implicit information, \emph{e.g.}\xspace \query{Find cars with maximum mpg} intends to perform data filtering to search for cars with the largest mpg value.
The use of a filter is identified by function classification in the query execution phase (\autoref{sec:function-classification}).

\section{Evaluation}
\label{sec:evaluation}

To evaluate the effectiveness of FlowSense\xspace, we describe the results of one case study and one formal user study.

\input{sections/evaluation/visionzero.tex}
\input{sections/evaluation/user-study}

\subsection{Speed Reduction Study}
We invite several users to try out the FlowSense\xspace prototype in different data analysis domains and analyze their usage of our NLI.
In this paper we introduce one case study in which we work with two domain experts in person to address a practical research task using a comprehensive set of NL queries.
The analysts are researching the city regulation issued on November 7, 2014
that reduces the default speed limit on all New York City streets from 30 MPH to 25 MPH.
The data contain the estimated average hourly speed~\cite{PocoJ15} for each road segment in Manhattan from January 2009 to June 2016.
The speed estimation was performed based on the TLC yellow taxi records~\cite{TLC} that only have pickup and dropoff information.
The analysts are familiar with the data, and the visualizations to be created are similar to the visualizations they previously generated for the project using Tableau~\cite{tableau}.
However they have no prior experience with either VisFlow\xspace or FlowSense\xspace.
We met the analysts in person and first introduced VisFlow\xspace and FlowSense\xspace in a 30-minute session.
Then we guided the analysts through how FlowSense\xspace can be used to create visualizations to study the speed reduction.
We observed in this study that almost all the analysts' visualization requests (excluding those that exceed the scope of the VisFlow\xspace subset flow) can be effectively supported by FlowSense\xspace.
Here we summarize the NL queries applied in the speed reduction study.

Initially, the analysts would like to look at the speed reduction impact at a larger scale.
They first load a pre-computed speed table (\autoref{fig:visionzero-a}(1)) with the FlowSense\xspace data loading utility function (the analysts know the dataset name).
The table contains the monthly average speed aggregated by the speed limits of the streets.
The analysts ask the system to present a histogram of speed by \query{Show speed distribution} (\autoref{fig:visionzero-a}(2)).
The first histogram has no color encoding but the analysts are able to immediately add a color scale by \query{Encode speed limit by color}.
FlowSense\xspace inserts a color mapping node with a red-green color scale at the input of the histogram (\autoref{fig:visionzero-a}(3)).
The histogram shows the street groups with higher speed limit in green, and lower speed limit in red.
To view the speed changes over time, the analysts use the query \query{Draw speed over time grouped by speed limit} (\autoref{fig:visionzero-a}(4)).
The query result is a line chart showing average speed changes for different speed limit groups.
The analysts observe that overall there is a speed reduction in all speed limit groups that started around middle 2013.

Seeing the overall trend, the analysts move on to a comparative analysis between individual streets from two slow zones.
They load and visualize a table about speed limit sign installation in a map (\autoref{fig:visionzero-b}(1)) by \query{Show the data in a map}.
This dataset has for each road segment in Manhattan its speed limit, geographical location, and whether the street has speed limit signs installed (signs are shown as dots in the map).
As the slow zones mostly have speed limit signs installed, the analysts narrow down the data in the map by placing a filter on the ``sign'' column (\autoref{fig:visionzero-b}(2)).
The filtered map reveals two slow zone neighborhoods with densely located signs: Alphabet City and West Village.
The analysts apply one map visualization for each zone for a comparison between the two zones.
They label the two maps by the slow zone names and select a few streets from each zone (marked in the maps of \autoref{fig:visionzero-b}).
To study the speed changes of these selected streets, another table (named ``segment monthly speed'', also known to the analysts) that includes monthly average speed for each road segment is added to the diagram (\autoref{fig:visionzero-b}(3)).
The analysts then use the link queries to create a sequence of nodes that extract segment IDs from the selected streets and find their monthly average speed from the segment monthly speed table (\autoref{fig:visionzero-b}(4)).
Blue and red colors are assigned to the streets in West Village and Alphabet City respectively to visually differentiate them (\autoref{fig:visionzero-b}(5)).
The two groups of streets are then merged by a subset manipulation function (\autoref{fig:visionzero-b}(6)).
Note that the query \query{Merge} only has a single word.
It works because the query completion of FlowSense\xspace automatically locates the recently focused color editors as the source nodes for this query.
Finally, the two groups are rendered together in a speed series visualization (\autoref{fig:visionzero-b}(7)), which compares the speed changes between the two groups of streets.
As the visualizations produced by FlowSense\xspace are linked,
the analysts can easily change the street selection in the maps to compare different groups of streets.

This case study demonstrates that FlowSense\xspace can be applied to a practical, comprehensive analytical task.
The generated visualizations may guide the analysts towards further data analysis.
The analysts participating in this study think FlowSense\xspace is helpful,
especially since it exemplifies how to build VisFlow\xspace diagrams and facilitates their learning of the DFVS.

\subsection{User Study}
\label{sec:user-study}

We conduct a formal user study to evaluate the effective of FlowSense\xspace together with the VisFlow\xspace framework.
Through the user study we validate whether a user is able to smoothly apply FlowSense\xspace for dataflow diagram construction, and how well FlowSense\xspace responses meet the user's expectation.
We design an experiment that introduces FlowSense\xspace and VisFlow\xspace to the participant and assigns analytical tasks to be solved within the system.

\subsubsection{Experiment Overview}

The user study is carried out in a fully automated manner using an online system with step-by-step instructions.
The participants join the study using a web browser on their own machines.
Participants may ask the experiment assistant for help and clarification via web chat or phone call during the experiment session.

We recruited $17$ participants ($11$ male, $6$ female, all with an age between $20$ and $30$) who work or study in the field of computer science.
$12$ participants have a data visualization background.
$9$ are graduate students, and the other $8$ are professionals (software engineer, researcher, faculty).
$3$ participants have prior experience with VisFlow\xspace.
No participants have prior knowledge about FlowSense\xspace.
The participants are chosen to have a variety of specialities so as to represent potential DFVS users.
The participant group includes visualization designers, data scientists, and software engineers who share data analysis interest but have different skill sets.
The study is structured into two phases:

\myparagraph{Tutorial Phase}.
The participant completes a tutorial of the VisFlow\xspace dataflow framework, and then a tutorial of the FlowSense\xspace NLI.
After each tutorial, the participant is asked to complete the tutorial diagram to demonstrate familiarity with the introduced tool.
Each tutorial is expected to take 10 to 20 minutes.
After the tutorials there is an on-demand practice session with a flexible duration.

\myparagraph{Task Phase}.
The participant explores an {\it SDE Test} dataset and constructs dataflow diagrams using FlowSense\xspace and VisFlow\xspace to answer questions about the data.
The participant is encouraged to use FlowSense\xspace as much as possible.
The usage of the NLI is not enforced because the goal of the NLI design is to improve the user experience of the DFVS, rather than to completely replace the traditional DFVS interactions (which is likely infeasible).
The entire task phase is expected to take 30 to 60 minutes.

At the end of the study, the participant takes a survey to give comments and quantitative feedback about FlowSense\xspace and VisFlow\xspace.

\subsubsection{Dataset and Tasks}

The {\it SDE Test} dataset includes the test results of software development engineer (SDE) candidates stored in two tables.
The first table describes the test results for each candidate.
A test consists of answering several multi-choice questions selected by the system from a large question pool.
Each question has a unique ID, a pre-determined difficulty, its supported programming language(s), and possibly a time limit.
For each question, the candidate receives a result (correct, wrong, skipped, unanswered)\footlabel{remarks}{See the appendix for additional remarks and results of the user study.}.
The dataset also has a ``TimeTaken'' column that stores how much time a candidate took to answer a question.
The second table includes background information about each candidate, such as the candidate's highest degree level, field of study, and institution.
We give three analytical tasks about this dataset.
The tasks are designed to reflect common tasks performed in visual data exploration:

\myparagraph{(T1) Overview Task}.
The participant is asked to visualize the overview distribution of the question answering results, and figure out the total number of questions that were skipped, and the percentage of a question being answered correctly.

\myparagraph{(T2) Outlier Task}.
The participant is first asked to find a candidate with an outlier background information value (who incorrectly entered the current year ``2018'' in place of his own information).
Then the participant is asked to investigate a data recording discrepancy regarding the ``TimeTaken'' column:
Some of the ``TimeTaken'' values are erroneously large numbers when a question is unanswered.

\myparagraph{(T3) Comprehensive Task}.
The participant is asked to identify one question that Masters candidates answer significantly better than Bachelors candidates.
This task requires comprehensive usage of the dataflow features: attribute filtering, brushing, and heterogeneous table linking.

All the three tasks have definitive correct answers to ensure that participants explore the data and draw conclusions reasonably.
Each user study session is logged with anonymous full diagram editing history.
We analyze the study results based on task answers and completion time, comments and quantitative feedback, and NL query logs.

\subsubsection{Task Completion Quality}

\begin{figure}
\centering
\subfigure[]{\label{fig:a}\includegraphics[width=.9\linewidth]{study/answers}}
\addtolength{\subfigtopskip}{-10pt}
\subfigure[]{\label{fig:b}\includegraphics[width=.9\linewidth]{study/times3}}
\vspace{-3mm}
\caption{
(a) Verdict distribution of participant answers to each of the user study tasks.
(b) Box plot of completion time for each user study step\protect\footref{remarks}.
}
\vspace{-5mm}
\label{fig:answers}
\end{figure}

\autoref{fig:answers}(a) shows the verdict distribution of the participants' answers.
It can be seen that the majority of the participants were able to come up with the correct answers to the tasks.
\autoref{fig:answers}(b) shows the completion time distribution for each step of the user study.

It can be observed that the time taken for the tutorials and tasks are mostly as expected.
Yet the time required for a task increases when the task involves heterogeneous tables and interactive data filtering to find solutions (T3).
After reading the user comments in the feedback, we believe this may be due to the fact that many participants are first-time VisFlow\xspace users and need to digest the concept of the VisFlow\xspace subset flow model.
In particular, linking heterogeneous tables can be challenging to understand at first.
However, most users were able to get the idea and formulate a solution.
This is reflected by one of the feedback comments:
\query{The linker functions are confusing at first. But after experimenting with the tool for a while and getting to know how they work, things become easier.}
We believe such a learning curve is natural for DFVS.

\subsubsection{Quantitative Feedback}

We ask for feedback on six aspects regarding FlowSense\xspace (and also VisFlow\footref{remarks}) in our survey.
Each aspect is presented with a statement and a $1$--$5$ Likert scale for the participant to express agreement ($5$) or disagreement ($1$).
Table~\ref{table:flowsense-survey} lists the feedback for the FlowSense\xspace NLI.
The quantitative feedback shows that most users were able to understand the scope of FlowSense\xspace,
and apply it for dataflow diagram construction.
The users were also asked to compare the NLI-assisted dataflow usage against their earlier experience in the tutorial phase with the standalone VisFlow\xspace framework.
Twelve users agree (with a feedback score of at least 4) that FlowSense\xspace simplifies the diagram construction, and ten users agree that FlowSense\xspace speeds up the data exploration.

The feedback also reveals space for improving the NLI.
In particular, it is unclear to most users how to update a rejected query to make it accepted.
It may be helpful to design an algorithm that provides suggested corrections or changes to a failed query.
However, this is technically challenging as changing minimally a query to fit it into the parse tree is algorithmically non-trivial.
We would like to leave query correction suggestions for future work.

\subsubsection{Query Log Analysis}

To closely study where FlowSense\xspace does not accept a query, we manually went over the rejected queries and categorized each rejected query by its reason of rejection.
Overall, we analyzed $649$ queries, out of which $421$ were accepted by FlowSense\xspace.
Excluding the $34$ invalid and mistyped queries, the raw acceptance rate was $68.455\%$.
We found some of the rejection issues straightforward to resolve: the requested functionality was not implemented, bugs in the query execution code, etc.
We were able to fix those issues in a short iteration of the NLI implementation, resolving $34$ ``not implemented'' queries and $18$ software bugs.
The improved acceptance rate would be 76.911\%.
In general, it requires systematic engineering efforts to thoroughly increase query coverage for the ``not implemented'' category, which is beyond the scope of this paper.
The remaining unresolved failures are summarized in \autoref{fig:reasons} with their counts\footnote{See the appendix for the detailed definition and examples for each category.}.

\input{sections/evaluation/flowsense-survey}

Some of those failures are more challenging to resolve.
Specifically, FlowSense\xspace does not make logical inferences and deals only with the raw values in the data.
If the user rephrases the query by natural language variation or implication (26 occurrences in~\autoref{fig:reasons}), the query would be difficult to parse.
The query \query{Show only segments with signs} is more natural than that in \autoref{fig:visionzero-b}(2).
Yet FlowSense\xspace does not infer that a segment with a ``sign'' value of ``yes'' implies that it is a segment ``with sign''.
In T3 the dataset has ``HighestLevelOfEducation'' as a column name, but if the user mentions ``degree'', FlowSense\xspace does not know that it is equivalent.
There needs to be additional knowledge base added to the system so that the NLI can detect concept equivalence, which is generally difficult to achieve.
In a ``composite'' query, the user intends to perform several VisFlow\xspace functions in one query (\emph{e.g.}\xspace creating nodes, applying filter, and assigning color together).
It is difficult to write concise grammar rules to accept composite queries.
In practice, by informing the users of these limitations, in most cases the issues can be circumvented via rephrasing the queries,
\emph{e.g.}\xspace composite queries can be split into smaller steps that are easier to parse and execute.

When an operation requested is not supported by the DFVS, a ``not supported'' failure arises,
\emph{e.g.}\xspace VisFlow\xspace without its data mutation extension\footnote{https://visflow.org/extension. The extension is not yet supported} cannot aggregate and mutate data.
When the special utterance tagging over-aggressively tags a non-special word, its placeholder fails to resolve, leading to a ``tagging error''.
The user may use the token dropdown in the FlowSense\xspace input box to correct tagging mistakes,
or disambiguate tokens with multiple meanings (\autoref{fig:flowsense-ui}(c)).

\begin{figure}[htbp]
\begin{center}
  \includegraphics[width=.95\linewidth]{study/reasons}
\end{center}
\vspace{-7mm}
\caption{Number of failed queries grouped by the reasons of their failures.
The colors of the bars indicate the relative difficulty of resolving a failure.}
\vspace{-3mm}
\label{fig:reasons}
\end{figure}

\section{Related Work}
\label{sec:related}

\subsection{Dataflow Visualization System (DFVS)}
Dataflow systems enable the user to configure system functionality by drawing a dataflow diagram that defines how the system modules interact with each other.
While dataflow systems are effective in fields other than data visualization such as computational workflow design~\cite{KatherineW13,SPSSModeler,Knime},
we focus on dataflow systems for visualization purposes in this section.
Previous DFVS have demonstrated the effectiveness of using dataflow to render scientific data~\cite{UpsonC89,ParkerS95,HaeberliPE88}
and manage volume rendering pipelines~\cite{BavoilL05,MeyerSpradowJ09}.
Dataflow systems that pass only data subsets (versus program method arguments) yield simpler dataflow diagrams and lower learning overhead~\cite{RobertsJC98, RobertsJC99}.
ExPlates~\cite{WaqasJ13} and VisFlow\xspace~\cite{YuB17} present embedded visualizations in their dataflow, and focus on interactive information visualization.
Most dataflow systems support diagram editing in a drag-and-drop manner.
However, it is observed that even with drag-and-drop interfaces, users may often have difficulty in translating their intention to system operations~\cite{GrammelL10}.
In this work we design FlowSense\xspace to further simplify dataflow diagram construction, so that the user can intuitively use dataflow and make the most of the analytical capability of a DFVS.
In particular, we build FlowSense\xspace for VisFlow\xspace, as its subset flow model supports many of the low-level visual data analysis tasks~\cite{ShneidermanB96, AmarR05}, such as characterizing distribution, finding extremum, etc.

\subsection{NLI for Data Visualization}
Extensive research has been devoted to NLIs for decades.
These interfaces address NL queries that otherwise have to be manually translated to formal query languages, \emph{e.g.}\xspace SQL.
A few examples are the interfaces for querying XML~\cite{YLi07}, entity-relational database~\cite{AndroutsopoulosI95, YinP16}, and sequence translator to SQL~\cite{ZhongV17}.
NLIs for data visualizations answer the queries by presenting visual data representations.
Compared with other interfaces that simply return a numerical answer or a set of database entries,
visualization NLIs present results that are more human-readable.
Cox et al.~\cite{CoxK01} design the Sisl service within the InfoStill data analysis framework.
The service asks a series of NL questions to complete an unambiguous query.
The Articulate system~\cite{SunY10} uses a Graph Reasoner to select proper visualizations to answer a query.
DataTone~\cite{TongG15} addresses query ambiguity by showing ambiguity widgets along with the main visualization so that the user is able to switch to desired alternative views.
Eviza~\cite{SetlurV16} and Evizeon~\cite{HoqueE18} further improve the user experience by allowing for conversation-like follow-up questions.
Fast et al.~\cite{FastE17} propose a conversational user interface called Iris that may perform analytical tasks and plot data upon requests in dialogues.
Kumar et al.~\cite{KumarA16} also propose a dialogue system for visualization.
Orko~\cite{SrinivasanA18} is an NLI designed for visual exploration of network data.
Dhamdhere et al.~\cite{DhamdhereK17} design Analyza that provides database-based NL query and visualizations.
Srinivasan et al.~\cite{SrinivasanA17} provide a summary and comparison of the majority of these NLIs.
Several commercial tools integrate NLIs.
IBM Watson Analytics~\cite{WatsonAnalytics} and Microsoft Power BI~\cite{PowerBI} provide a list of relevant data and visualizations to an NL question, from which the user may choose to continue an analysis.
Wolfram Alpha~\cite{WolframAlpha} supports knowledge-based Q\&A and is able to plot the results.
ThoughtSpot~\cite{ThoughtSpot} enables interactive search in a relational database, and provides multiple types of visualizations for the database.
The NLI design for data visualization has two challenges:
First, modern natural language processing (NLP) techniques cannot yet understand well arbitrary NL input due to the complex nature of NL.
User queries are apt to be free-form and ambiguous;
Second, choosing a proper visualization to answer an analytical question is non-trivial as there can be multiple possible visual representations~\cite{MackinlayJ07}.

\subsection{Comparison with Other NLIs}

\noindentFlowSense\xspace makes a distinction from the other interfaces as it is to our best knowledge the first NLI to address a dataflow context.
We set the scope of FlowSense\xspace to focus on assisting dataflow diagram construction, rather than to directly answer free-form analytical questions or seek a best visualization for a given query.
We believe such an approach is beneficial in several aspects:\\
\myparagraph{Capability:} The analytical capability of FlowSense\xspace is rooted in the design of the DFVS.
The outcome of FlowSense\xspace is a complete, interactive, and iterative visual data exploration process supported by the DFVS, rather than a single visualization that only answers one particular query as in many other interfaces.
Dataflow also naturally preserves analysis provenance~\cite{FreireJ06},
allowing the user to frequently revisit and reassess the current workflow.
The diagram created by FlowSense\xspace explicitly keeps the user's preference and intention from previous queries,
which must otherwise be maintained by a model behind the scene~\cite{TongG15,SetlurV16}.

\myparagraph{Usability:}
FlowSense\xspace integrates real-time presentation of tagged special utterances in the interface that reflect the state of the underlying semantic parser and help the user understand the dataset and dataflow present in the system (\autoref{sec:dataflow-context}).
This is a novel design that facilitates the user's understanding of the NLI behavior, as in most other NLIs the parsing feedback is only given after the query is submitted.
The auto-completion suggestions of FlowSense\xspace also present special utterance tags so that the user may better understand the expected query components.
Consequently, FlowSense\xspace may ease DFVS usage and make DFVS more accessible.
Our case study and user study (\autoref{sec:evaluation}) show that FlowSense\xspace improves the DFVS usability, and its convenience is desirable by both novice and experienced VisFlow\xspace users.
Besides, the DFVS is able to recover from errors more easily as the user always has full control over the system.
However in other interfaces the user has to mostly rely on the behavior of the NLI and can hardly make corrections in case of misinterpretation.

\myparagraph{Feasibility:}
The scope of assisting dataflow diagram construction is well defined and practicable.
Even state-of-the-art NLP techniques have limited success in understanding an arbitrary query.
Because each query is expected to update dataflow diagram and the user decides what the system should do and what visual representation to apply,
FlowSense\xspace can produce more expected results and give better user experience under a well-defined scope.
The mixed-initiative design mitigates the ambiguity problem.
The DFVS users in our case study and user study are all able to understand the scope of FlowSense\xspace and use FlowSense\xspace effectively.

\input{sections/functions-table}
\input{sections/fig-pattern}

\subsection{Semantic Parsing}
FlowSense\xspace uses semantic parsing to process NL input and map user queries to VisFlow\xspace functions (\autoref{sec:functions}).
It depends on a pre-defined grammar that captures NL input patterns.
A semantic parser recursively expands the variables in the grammar to match the input query and can interpret the input based on the rules applied and the order of their application~\cite{BerantJ13}.
At a high level, the mapping performed by FlowSense\xspace can also be considered a classification task and addressed by classification algorithms~\cite{AllahyariM17}.
However we prefer semantic parsing because most classification approaches are supervised algorithms that require a large corpus of labeled examples.
Such training data are not available for DFVS.
Besides, compared with deep learning methods~\cite{LiD14,GoodfellowI16}, semantic parsing does not require heavy computational resources.

The FlowSense\xspace semantic parser is implemented within the Stanford SEMPRE\xspace framework~\cite{PasupatP15} and CoreNLP toolkit~\cite{ManningCD14}.
The CoreNLP toolkit integrates a comprehensive set of NLP tools including the Part-of-Speech (POS) tagger, Name-Entity-Recognizer (NER), etc.
A POS tagger identifies roles of words in a sentence, e.g. verb, preposition, adverb.
The SEMPRE\xspace framework employs a modular design in which different types of parsers and logical forms can be easily plugged-in.
The framework can quickly be adapted for domain-specific parser design~\cite{WangY15}.
We apply SEMPRE\xspace together with CoreNLP to the DFVS domain.
In particular, the FlowSense\xspace parser utilizes the POS tags produced by CoreNLP for processing special utterances and grammar matching.
The FlowSense\xspace grammar expects words with certain POS tags to appear in query parts.

\section{Conclusions}
\label{sec:conclu}

In this work we design FlowSense\xspace, a novel NLI for visual data exploration within a DFVS.
We build FlowSense\xspace for the VisFlow\xspace framework and show that it improves the DFVS usability and simplifies diagram construction.
FlowSense\xspace applies semantic parsing to map NL input to VisFlow\xspace functions.
Its emphasis on special utterances and usage of special utterance placeholders make the semantic parsing independent of datasets and diagrams, but at the same time aware of the dataflow context.
The real-time feedback of tagged special utterances, as well as query and token auto-completion features, largely helps the user understand the underlying parsing state.
Our case study and user study results demonstrate the effectiveness of the proposed NLI, and help identify future research directions for its improvement.

\section{FlowSense Grammar Design}
We provide an open source repository that contains the details of the FlowSense\xspace implementation: \url{https://github.com/yubowenok/flowsense}.
This repository includes the grammar rules, backend API (implemented in TypeScript and Python), and integration tests.
The structure of this repository and its installation and setup guide can be found within its \href{https://github.com/yubowenok/flowsense/blob/master/README.md}{README} file.

In particular, the grammar rules are located in the \href{https://github.com/yubowenok/flowsense/tree/master/src/grammar}{{\tt *.grammar} files}.
The entry point is {\tt main.grammar}.
The grammar rules are written in the SEMPRE\xspace grammar format (\emph{i.e.}\xspace SEMPRE formulas).
More details on SEMPRE can be found at the \href{https://github.com/percyliang/sempre}{SEMPRE\xspace GitHub repository}.

\section{Characteristics of the Diagram Editing Focus Heuristics}
Intuitively, the focus score keeps track of the diagram element that is last interacted with.
It has two components: the activeness resulted from mouse clicks, and the distance-to-mouse bonus.
The activeness score exponentially decreases when there is no interaction on the node, while the distance-to-mouse bonus prioritizes the elements around the last interaction.

When the mouse hits a node $x$, node $x$ receives a high activeness score of one from the {\it Click(X, t)} part,
which almost certainly ensures that the focus score of $x$ is higher than any other node $y$ that is not interacted with.
Though $y$ (when it is in the proximity of $x$) may receive a distance-to-mouse bonus that remedies its exponential loss on the activeness score,
note that $x$ receives a distance-to-mouse bonus too, and the bonus can only be higher than the bonus received by $y$ because $x$ is clicked on and thus closer to the mouse.
Therefore, the outcome is that $x$ becomes the first prioritized node, and $y$ becomes the second prioritized.
In other words, if a VisFlow\xspace function requires two node operands, then $x$ is chosen first, and then $y$ is chosen.

If the user clicks on the background, all nodes have exponentially decreasing activeness score,
and their distance-to-mouse bonus will likely dominate the focus score.
Consequently, the nodes that are closer to the last click become the chosen query targets.
As there can be multiple nodes around the background click,
occasionally a node not actually focused by the user may happen to be close to an unintentional background click (\emph{e.g.}\xspace accidentally performed during canvas panning).
The next NL query may then be incorrectly performed on this node.
This error can be fixed by clicking on a specific node to focus on it and redoing the NL query.

\section{Additional User Study Remarks}
\begin{enumerate}[*]
\item In the SDE test, answering a question wrong results in negative score penalty.
Therefore skipping a question can be worthy.
Skipping requires an explicit button click.
The ``unanswered'' result is given when the user has no action within the allocated time limit of a question.
\item In Fig. 6(b), four outliers due to interruptions on the participant's end are not shown: 2550 minutes on Task1,  and 109, 119, 212 minutes on Task3 were measured as the task completion time that includes the interruptions.
\end{enumerate}

\input{visflow-survey}

\section{VisFlow Survey Results}

The VisFlow\xspace user study is part of the FlowSense\xspace user study.
Before FlowSense\xspace was introduced to the users, a tutorial on the details of the VisFlow\xspace dataflow framework was given.
The participant group of the VisFlow\xspace user study is thus exactly the same as the group described in the paper:
In total, $17$ users ($11$ male, $6$ female, all with an age between $20$ and $30$) who work or study in the field of computer science participated in this study.
$12$ participants have a data visualization background.
$9$ are graduate students, and the other $8$ are professionals (software engineer, researcher, faculty).
$3$ participants have prior experience with VisFlow\xspace, who may yet formally evaluate it through task completion.
The participants are chosen to have a variety of specialities so as to represent potential DFVS users.
The participant group includes visualization designers, data scientists, and software engineers who share data analysis interest but have different skill sets.

The users were given a form to assess the effectiveness of VisFlow\xspace quantitatively using a Likert scale of 1 to 5 (5 is ``strongly agree'' and 1 is ``strongly disagree'').
\autoref{table:visflow-survey} shows the quantitative survey feedback for the VisFlow\xspace DFVS.
It can be observed that the users were able to understand the subset flow model of VisFlow\xspace.
The majority of the users agree (with a score of at least 4) that VisFlow\xspace presents an effective approach to visual data exploration,
and can successfully utilize VisFlow\xspace features for their data exploration.

\section{Query Analysis -- Failure Category Description}

This following list provides the description of each query failure category we identified in the user study results:

\begin{itemize}
\item \myparagraph{Not Implemented}.
FlowSense\xspace grammar may technically support parsing this query.
Yet we have not implemented the corresponding grammar and its web client handler (query execution code for diagram update).
Example queries include \query{change the x column to mpg}.
The current system implementation does not support node option changes triggered by the NLI (except for visual editors).
Queries of this category can be accepted by extending the grammar and adding more rules.

\item \myparagraph{Invalid/Mistyped}.
The query is an invalid sentence and cannot be understood by a human;
Or the query has mistyped words and fails to describe the intended data entity or dataflow element.

\item \myparagraph{Rephrased}.
The user rephrases the query using grammatical structures not expected by the grammar,
or the user uses words that do not appear in the dataset table to describe a table column or cell value.
For example, in Task 3 if the user mentions ``degree'', FlowSense\xspace does not know that ``degree'' is equivalent to the ``HighestLevelOfEducation'' column in the data.
Though one can inform a system of such equivalence case-by-case or find synonyms from WordNet,
it is non-trivial to generally detect such equivalence.
Consider the equivalence between \query{Show only segments with signs} and \query{Show only segments with a sign of yes}:
``yes'' is not an immediate synonym of ``with'' and their common implication of ``existence'' is subtle.
There needs to be additional knowledge base added to the system to support the detection of concept equivalence.

\item \myparagraph{Not Supported}.
The functionality indicated by the query is not supported by the VisFlow\xspace dataflow framework.
This is not an issue of the NLI but a limitation of the underlying DFVS.
A query like \query{How many questions were skipped} asks directly an analytical question about the dataset and exceeds the scope of VisFlow\xspace.
It cannot be accepted by simply extending the grammar because there needs to be a reasonable way to construct dataflow sub-diagrams to answer the analytical questions,
which can be complex and challenging to identify.

\item \myparagraph{Tagging Error}.
A special utterance should have (have not) been tagged, but it was not (was) tagged.
For example, the query \query{Select iris with id between 3 and 5} has the word ``iris'' that is both a word to describe the data entity and a dataset name.
When FlowSense\xspace automatically tags ``iris'' as a dataset name special utterance, the parser may fail to accept the query.
In this case the user may manually override the tagging to avoid the error resulted from parsing ambiguity.
In future work we may also explore techniques that can be integrated into the parser to structurally reduce such errors.

\item \myparagraph{Composite}.
The user inputs a query that attempts to execute multiple VisFlow\xspace functions that exceed the limit expected by the grammar or the web client handler.
An example is \query{Highlight bachelors in red and masters in green in node-15}.
This is achievable using attribute filters to find candidates with Bachelors and Masters degrees, followed by visual editors to give them colors, and finally a set operator to merge the two groups.
In this case multiple VisFlow\xspace functions have to be performed altogether.
The parser and execution handler did not expect queries of this composite level.
The grammatical structure between these multiple functions poses parsing difficulty.
It is recommended that composite queries are split into multiple smaller steps so as not to overload the NLI with a complicated grammatical structure that exceeds its parsing capability.

\item \myparagraph{Bug}.
The system should have the capability of handling that query.
But due to an implementation bug that was unidentified at the time of the user study,
the query parsing or execution went wrong and did not arrive at the expected outcome.

\end{itemize}

\end{document}