\section*{Checklist}

\begin{enumerate}

\item For all authors...
\begin{enumerate}
  \item Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
   \answerYes{}
  \item Did you describe the limitations of your work?
    \answerYes{}
  \item Did you discuss any potential negative societal impacts of your work?
    \answerYes{}
  \item Have you read the ethics review guidelines and ensured that your paper conforms to them?
    \answerYes{}
\end{enumerate}

\item If you are including theoretical results...
\begin{enumerate}
  \item Did you state the full set of assumptions of all theoretical results?
    \answerNA{}
        \item Did you include complete proofs of all theoretical results?
    \answerNA{}
\end{enumerate}

\item If you ran experiments...
\begin{enumerate}
  \item Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
    \answerNo{}
  \item Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
    \answerYes{}
        \item Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
    \answerNo{}
        \item Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
    \answerNo{}
\end{enumerate}

\item If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
\begin{enumerate}
  \item If your work uses existing assets, did you cite the creators?
    \answerYes{}
  \item Did you mention the license of the assets?
    \answerYes{}
  \item Did you include any new assets either in the supplemental material or as a URL?
    \answerNo{}
  \item Did you discuss whether and how consent was obtained from people whose data you're using/curating?
    \answerYes{}
  \item Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
    \answerYes{}
\end{enumerate}

\item If you used crowdsourcing or conducted research with human subjects...
\begin{enumerate}
  \item Did you include the full text of instructions given to participants and screenshots, if applicable?
   \answerNA{}
  \item Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
    \answerNA{}
  \item Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
    \answerNA{}
\end{enumerate}

\end{enumerate}

\section{Tips}

\input{tex/intro}
\input{tex/related}
\input{tex/method.tex}

\input{tex/exp}

\section{Discussion}

\myparagraph{Conclusion.} In this paper, we propose 3D-CG, which leverages the continuous and differentiable nature of neural descriptor fields to segment and learn concepts in the 3D space. We define a set of neural operators on top of the neural field, with which not only can semantic and instance segmentations emerge naturally, but visual reasoning can also be well performed.

\myparagraph{Limitations and Future Work.} An limitation of our underlying framework with 3D-CG is that while we show transfer results on real scenes, our approach is only trained with synthetic scenes and questions. While we believe our proposed operations is general-purpose, an interesting direction of future work would be scale our framework to directly train on complex real-world scenes, and ground more concepts from natural language on these real scenes is worth delving into in the future.
\begin{ack}
This work was supported by MIT-IBM Watson AI Lab and its member company Nexplore, ONR MURI, DARPA Machine Common Sense program, ONR (N00014-18-1-2847), and Mitsubishi Electric.
\end{ack}

\bibliographystyle{abbrv}

\section*{Overview}
In this appendix, we supplement our paper by providing more qualitative examples and details about our model to help readers better understand our paper.

This appendix is organized as follows.
\begin{itemize}
    \item In Sec.~\ref{3d-cg}, we provide more qualitative examples about how 3D-CG performs reasoning steps.
    \item In Sec.~\ref{details}, we provide more details about our 3D-CG model, including the detailed architecture of NDF, the semantic parsing module and the curriculum learning strategies.
    \item In Sec.~\ref{baselines}, we provide more details about the baselines models.
    \item In Sec.~\ref{impacts}, we discuss the potential societal impacts of this paper.
\end{itemize}

\newpage
\section{More Qualitative Examples}
\label{3d-cg}
\label{examples}
\subsection{QA Examples}
In Figure \ref{fig:qaexamples}, we show more examples of the reasoning process of 3D-CG. We can see that overall, our model can make reference to the correct region to answer the questions, although sometimes the segmentation might be a little noisy (\textit{e.g.,} for the bag a part of the shoulder strap is not highlighted, and the highlighted part of cart contains part of the wheel.). Also, for counting problems, we can see that our model can segment the legs and make the right prediction. However, for the chair with five legs, although our model can assign five colors to five legs, some of the legs are overlapped with each other by colors.
\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.8\textwidth]{fig/supp_reference.pdf}
    \caption{More qualitative examples that illustrate the reasoning process of 3D-CG. Given an input
image of a shape and a question, we first parse the question into steps of operators. We visualize the set of points
being referred to by the operator via highlighting the regions where mask probabilities > 0.5. As is shown,our
3D-CG can make reference to the right set of coordinates, thus correctly answering the questions.}
    \label{fig:qaexamples}
\end{figure}

\newpage
\section{Details on 3D-CG}
\label{details}
\subsection{NDF}
We first train the reconstruction task solely for 50,000 iterations.

For encoder, we use a PointNet-based encoder network with ResNet blocks. We first map the 6-dim (x,y,z,r,g,b) point cloud to 128-dim features, followed by several pooling and leaky relu layers. We then go through another fully-connected layer to get 256-dim features. We use 5 resnet fully connected blocks, each with input dim 256 and output dim 128. At each step the output is concatenated with the input to produce the input of the next step. A final fully-connected layer maps the feature back to 128.

For the decoder, we first concatenate the coordinates and the point cloud latent feature vector. The new input is passed through 5-layer resnet blocks. All the resnet blocks have a hidden size of 128. The features of the blocks are concatenated together and passed through a fully-connected layer to produce the occupancy value. For color reconstruction, the features further go through 3 linear layers with hidden size 128 and produce r,g,b value. The batch size is 4. The learning rate is 1e-4. All embeddings have the dim 2049.

\subsection{Semantic Parsing}
Our semantic parser is an attention-based sequence to sequence (seq2seq) model with an encoder-decoder structure similar to that in \cite{Luong2015EffectiveAT}. The encoder is a bidirectional LSTM \cite{Hochreiter1997LongSM}. The decoder is a similar LSTM
that generates a vector from the previous token of the output sequence. Both the encoder and decoder
have two hidden layers with a 256-dim hidden vector. We set the dimensions of both the encoder and
decoder word vectors to be 300.

\subsection{Curriculum learning}
\begin{wrapfigure}{r}{0.35\linewidth}
\centering
\includegraphics[width=\linewidth]{fig/supp_curriculum.pdf}
\caption{The curriculum learning strategy of 3D-CG.}
\label{fig:curriculum}
\end{wrapfigure}

In Figure \ref{fig:curriculum}, we show detailed strategies of curriculum training for 3D-CG. During the whole training process, we gradually add more visual concepts and more complex question examples into the model, which is shown in Figure \ref{fig:curriculum}. In general, the whole training process is split into 3 stages. First, we learn \texttt{color} concepts. We want to learn individual concepts like \texttt{red} or \texttt{seat}. However, since most chairs contain certain parts like \texttt{back} and \texttt{seat}, we therefore learn the \texttt{color} concepts first by asking questions like ``Is there a red part in the chair?" For the second lesson, after learning about \texttt{color} concepts like in the figure, we can then attend to the red region, and we can ground different part categories by asking questions like ``What is the category of the red part?". Finally, we can train more complex questions altogether. Each lesson is trained for approximately 50,000 iterations.

\newpage
\section{Datails on Baseline Models}
\label{baselines}
\subsection{Details on Language-mediated models}
                                                                                We explain in detail how CVX-L and BAE-L work for segmentation and reasoning, as is shown for Figure \ref{fig:bae}. As the original CVX/BAE model, we use the same NDF as in 3D-CG, except that instead of decoding the descriptor vectors into N*1 occupancy values, we first decode them into N*S occupancy values for different slots, and then maxpool to get the overall occupancy values for reconstruction. For reasoning, the descriptor vectors of each slot is mean pooled to get a part-centric representation. The question also goes through a semantic parsing module like 3D-CG, and  attention is calculated between the concept embedding vector and the descriptor vector of each slot to identify whether the part in one slot can be accounted for a concept. The reasoning loss can also be back-propagated to the NDF module and the concept embeddings. We use the same curriculum learning strategy as 3D-CG and train for the same number of epochs.
\begin{figure}[htbp]
    \centering
    \includegraphics[width=\textwidth]{fig/supp_bae.pdf}
    \caption{Detailed framework of CVX-L / BAE-L, where N denotes the number of sampled points, and S denotes the number of slots.}
    \label{fig:bae}
\end{figure}

\subsection{Neural-network based baselines}
For MAC, we use an ImageNet-pretrained ResNet-101 to extract 14 × 14 × 1024 feature maps for MAC. For PointNet+LSTM, we use a 2048 dimensional feature
from the last layer. For NDF+LSTM, we mean-pool the descriptor vectors of all point coordinates to get 2049-dimensional vectors. All models are trained for 50 epochs.

\newpage

\section{Societal Impacts}
\label{impacts}
Our work is of broad interest to the computer vision and machine learning communities. Our proposed no inherent ethical or societal issues on its own, but inherit those typical of learning methods, such as capturing the implicit biases of data. Our work aims to enable more interpretable reasoning, and may serve as an inspiration for works that aims to perform less black-box reasoning with gradient-based learning.

\section{Experiments}
\subsection{Experimental Setup}
\subsubsection{Dataset}
Instead of object-based visual reasoning tasks where objects are spatially disentangled, which makes segmentations quite trivial, we seek to explore part-based visual reasoning tasks where segmentations and reasoning are both harder. To this end, we collect a new dataset, \texttt{PartNet-Reasoning}, which focuses on visual question answering on the \texttt{PartNet} \citep{Mo2019PartNetAL} dataset. Specifically, we render approximately 3K RGB-D images from shapes of 4 categories: \texttt{Chair, Table, Bag} and \texttt{Cart}, with 8 questions on average for each shape. We have three question types: \texttt{exist\_part} queries about whether a certain part exists by having the filter operator as the last operator; \texttt{query\_part} uses query operator to query about an attribute (\textit{e.g.}, part category or color) of a filtered part; \texttt{count\_part} utilizes the count operator to count the number of instances of a filtered part. We are interested in whether the reasoning tasks can benefit the segmentations of the fine-grained parts, as well as whether the latent descriptor vectors by NDF can result in better visual reasoning.

\subsubsection{Evaluation Tasks}

\vspace{5pt}

\myparagraph{Reasoning.} We report the visual question answering accuracy on the \texttt{PartNet-Reasoning} dataset w.r.t the three types of questions on all four categories.

\myparagraph{Segmentation.} We further evaluate both the performances of semantic segmentation and instance segmentation on our dataset.
As stated in \ref{sec:segmentation}, semantic segmentation can be performed by querying the part category with the maximum mark probability of each coordinate, and filtering the coordinates according to category labels, and instance segmentation can be achieved by using the count operator on each part category.
We report the mean per-label Intersection Over Union (IOU).

\subsubsection{Baselines}
\begin{table*}[]
    \centering
    \small
    \begin{tabular}{cc|cccccc}

         &&PointNet+LSTM & MAC & NDF+LSTM & CVX+L & BAE+L &3D-CG \\
         \hline
         \multirow{3}{*}{Chair} & exist\_part&52.3&65.2&55.7&71.9&72.4&\textbf{85.4} \\
    & query\_part &41.6&53.9&54.2&\textbf{72.3}&70.5&68.7\\
        & count\_part &63.4&78.1&71.6&48.8&68.0&\textbf{92.2}\\\hline
         \multirow{3}{*}{Table}& exist\_part &55.1&66.4&61.3&68.5&71.0&\textbf{80.3}\\
        & query\_part &43.6&51.2&54.4&69.7&73.6&\textbf{75.1}\\
        & count\_part &35.5&55.3&50.1&30.7&45.7&\textbf{90.9}\\\hline
         \multirow{3}{*}{Bag}& exist\_part &65.4&85.4&69.2&87.3&73.8&\textbf{89.2}\\
        & query\_part &53.1&74.8&64.0&\textbf{88.1}&70.6&85.2\\
        & count\_part &51.2&70.9&\textbf{72.3}&53.4&31.4& 68.3\\ \hline
         \multirow{3}{*}{Cart}& exist\_part &49.7&75.3&61.7&79.1&\textbf{91.0}& 90.1\\
        & query\_part &50.1&64.0&57.0&72.3&81.5& \textbf{86.3}\\
        & count\_part &41.6&\textbf{82.1}&67.3&46.6&47.3& 74.8\\\hline
    \end{tabular}
    \caption{\small \textbf{Visual question answering accuracies.} \texttt{exist\_part} queries about whether a certain part exists by having the filter operator as the last operator; \texttt{query\_part} uses query operator to query about an attribute of a filtered part; \texttt{count\_part} utilizes the count operator to count the number of instances of a filtered part. Point+LSTM, MAC and NDF+LSTM are methods based on neural networks. CVX+L and BAE+L are neural-symbolic methods which pre-segment the masks and ground concepts on the masks. 3D-CG outperforms baseline models by a large margin.}
    \label{tab:reasoning}
\end{table*}

We compare our approach to a set of different baselines, with details provided in the supp. material.

\myparagraph{Unsupervised Segmentation Approaches.} We compare 3D-CG with two additional 3D unsupervised segmentation models, and further consider how we may integrate such approaches with concept grounding.
First, we consider \textbf{CVXNet} \citep{Deng2020CvxNetLC}, which decomposes a solid object into a collection of convex polytope using neural implicit fields. In this baseline, a number of primitives are selected and a convex part is generated in each primitive. Next, we consider \textbf{BAENet} \citep{Chen2019BAENETBA} which decomposes a shape into a set of pre-assigned branches, each specifying a portion of a whole shape. In practice, we tuned with the original codes of BAENet on our dataset and found that nearly all coordinates would be assigned to a single branch. This is probably because the original model is trained on voxel models, while ours is on partial point cloud and contains more complex shapes, thus posing more challenges. Therefore, for BAENet we use an additional loss which enforces that at least some of the branches should have multiple point coordinates assigned to positive occupancy value. The primitives in CVXNet and the branches in BAENet are similar to slots in our paper.

\myparagraph{Reasoning Baselines.} We compare our approach to a set of different reasoning baselines. First we consider \textbf{PointNet+LSTM} and \textbf{NDF+LSTM}, which use the features by PointNet or our NDF module, concatenated with the features of the questions encoded by a LSTM model. The final answer distribution is predicted with an MLP. Next, we compare with \textbf{MAC}, a commonly-used attention-based model for visual reasoning. We add a depth channel to the input to the model. Finally we compare with \textbf{CVX+L} and \textbf{BAE+L} are neural-symbolic models that use the similar language-mediated segmentation framework from \cite{Wang2021LanguageMediatedOR}. Specifically, they first initiate the segmentations in the slots, and each slot has a slot feature vector. The operators from the questions are executed symbolically on the slot features. Question answering loss can be also propagated back to finetune the slot segmentations and features.

\myparagraph{Segmentation Baselines.}
For the segmentation tasks, we compare our approach with unsupervised segmentation approaches \textbf{CVX} and \textbf{BAE} described above. We further construct language-mediated variants \textbf{CVX-L} and \textbf{BAE-L}. For semantic segmentation by CVX and BAE, we use the manual grouping methods as in \cite{Deng2020CvxNetLC}. For semantic segmentation by language-mediated models, we use the filter operator to filter the slots belonging to the part categories.

\subsection{Experimental Results}
\subsubsection{Reasoning \& Concept Grounding}
\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{fig/reference_v2.pdf}
    \caption{\small \textbf{Qualitative Illustration.} Examples of the reasoning process of our proposed 3D-CG. Given an input image of a shape and a question, we first parse the question into steps of operators. We visualize the set of points being referred to by the operator via highlighting the regions where mask probabilities > 0.5. As is shown,our 3D-CG can make reference to the right set of coordinates, thus correctly answering the questions.}

    \label{fig:reference}
\end{figure}

Table \ref{tab:reasoning} shows the VQA accuracy among all models on our dataset. We can see that overall, our 3D-CG outperforms baseline modeles by a large margin. Language-mediated neural-symbolic methods (CVX+L \& BAE+L) are better than neural methods in the \texttt{exist\_part} and \texttt{query\_part} question types, but in general they are far worse than our method. This is because the slot-based representations pre-select the parts and make it easier for the concepts to attend to specific regions. However, wrong pre-segmentations are also hard to be corrected during the training of reasoning. The accuracies for the \texttt{count\_part} type further demonstrates this point, where language-mediated baselines have extremely low accuracies, and segmentations from Figure \ref{fig:segmentation} also show that even with supervision from question answering loss, BAE and CVX cannot segment the instances of a part category (\textit{e.g.}, legs and wheels). Neural methods such as MAC and NDF+LSTM perform well in the \texttt{count\_part} type in some categories,
probably due to some shortcuts of the \texttt{PartNet} dataset (\textit{e.g.}, most bags have two shoulder straps). In comparison to both neural and neural-symbolic methods, our method performs well in all three question types. This is because the regions attended by concepts are dynamically improved with the question answering loss, while the advantage of explicit disentanglement of concept learning and reasoning process is maintained.
\begin{figure}[t!]
\vspace{-4mm}
    \centering
    \begin{subfigure}{.67\textwidth}
    \includegraphics[width=\linewidth]{fig/visualization_v3.pdf}
    \caption{\small \textbf{Visualization of segmentation results.} }
    \label{fig:segmentation}
    \end{subfigure}\hspace{0.07\textwidth}%
    \begin{subfigure}{.18\textwidth}
    \includegraphics[width=\linewidth]{fig/generalization.pdf}
    \caption{\small \textbf{Generalization.} }
    \label{fig:generalization}
    \end{subfigure}
\caption{Visualization of segmentation as well as generalization. CVX and BAE are unsupervised segmentaton methods. CVX+L and BAE+L are language-mediated methods which utilize question answering loss to finetune the segmentation module. 3D-CG has better performances in both semantic and instance segmentations, while other methods suffer from merging parts or segmenting a part into multiple parts. It can also generalize well to unseen categories and real scans.}

\end{figure}

Figure \ref{fig:reference} shows some qualitative examples of the reasoning process of our method. Specifically, the questions are parsed into a set of neural operators. For each step, the neural operator takes the output of the last step as its input. For the filter operator, we visualize the attended regions with mask probabilities greater than 0.5. From the figure, we can see that our method can attend to the correct region to answer the questions, as well as segment the instances correctly for counting problems. This attention-based reasoning pipeline is also closer than previous neural-symbolic methods to the way that humans perform reasoning. For example, when asked the question ``What is the category of the green part of the chair?", humans would directly pay attention to the green region regardless of the segmentations of the rest of the chair. However, previous neural symbolic methods \citep{Yi2018NeuralSymbolicVD, Mao2019TheNC, Wang2021LanguageMediatedOR} segment the chair into different parts first and then select the green part, which is very unnatural.

\subsubsection{Segmentation}

Table \ref{tab:segmentation} shows the semantic segmentation and instance segmentation results, and Figure \ref{fig:segmentation}
visualizes some qualitative examples. We can see that overall, our 3D-CG can better segment parts than unsupervised or language-mediated methods. Other methods experience some common problems such as failing to segment the instances within a part category (\textit{e.g.}, one of the legs is always merged with the seat), or experience unclear boundaries and segment one part into multiple parts (\textit{e.g.}, segmenting the chair back or the tabletop into two parts). In general, language-mediated methods are better than pure unsupervised methods, which indicates that the natural supervision of question answering does benefit the segmentation modules. However, a lot of the wrong segmentations cannot be corrected by reasoning, resulting in bad results for counting questions for these models. In contrast, our 3D-CG can learn segmentations well using question answering loss, mainly because the segmentations emerge naturally from scratch without any restrictions.

\subsubsection{Generalization}
In Figure \ref{fig:generalization}, we show some qualitative examples of how our 3D-CG trained on seen categories can be generalized directly to unseen categories and real scans. We show results of semantic and instance segmentations, as well as visualize the parts that are referred to in the questions.
\begin{table}[t]
\begin{subtable}{0.49\textwidth}
    \centering
    \resizebox{\linewidth}{!}{
    \begin{tabular}{c|cccc|c}
         & {Chair} & {Table} & {Bag} & {Cart}&Mean\\
         \hline
        CVX&62.3 &74.2&66.0 &44.2 & 61.7\\
        CVX+L&64.6 & 74.5&\textbf{70.2} &51.3 &65.2 \\
        BAE & 49.5 & 71.0&64.1 &49.7 &58.6\\
        BAE+L & 56.3 & 72.3&69.8 &46.9 & 61.3\\
        3D-CG & \textbf{76.6} & \textbf{79.3}&67.3 & \textbf{54.2} & \textbf{69.4}\\
    \end{tabular}}
    \subcaption{\small Semantic Segmentation IOU}
    \label{tab:semantic}
    \end{subtable}%
\hfill
\begin{subtable}{0.49\textwidth}
    \centering
    \resizebox{\linewidth}{!}{
    \begin{tabular}{c|cccc|c}
         & {Chair} & {Table} & {Bag} & {Cart}&Mean\\
         \hline
        CVX&44.1 &40.6&31.2 &22.5 & 34.6\\
        CVX+L&42.6 & 51.2&41.3 &20.4 &38.9 \\
        BAE & 53.3 & 32.2&39.8 &34.0 &39.9\\
        BAE+L & 50.1 & 47.0&34.1 &36.6 & 42.0\\
        3D-CG & \textbf{68.5} & \textbf{71.2}&\textbf{47.2} &\textbf{40.5} & \textbf{56.9}\\
    \end{tabular}}
    \subcaption{\small Instance Segmentation IOU}
    \label{tab:instance}
\end{subtable}
\hfill
\caption{\textbf{Segmentation Results}. 3D-CG outperforms all unsupervised / language-mediated baseline models.}
\label{tab:segmentation}
\vspace{-7mm}
\end{table}

For generalizing to new categories, we use the model trained on carts to ground and count the instances of the concept ``wheel" in cars. We can see that the instance segmentation results on the wheels are not perfect because one wheel is in wrong position. However, most of the wheels are detected and the model manages to output the right count. We also use the model trained on chairs to filter the legs of a bed. All the legs are successfully selected out by our 3D-CG.

We also use real 3D scans from the RedWood dataset \citep{Choi2016ALD} to estimate 3D-CG's ability to generalize to real scenes. We use a single-view scan to reconstruct partial point cloud and remove the background such as the floor. We input the partial point cloud into our 3D-CG and output the segmentations on the ground-truth voxels. For generalizing to bicycles, we use the 3D-CG model trained on carts. We can see that 3D-CG can find all instances in the bicycle and detect both wheels. Furthermore, 3D-CG trained on chairs can also be generalized to chairs in real scans.

\section{Introduction}

Consider an image of a table pictured in Figure \ref{fig:teaser}. We wish to construct a method that is able to accurately ground the concepts and reason about the image such as the number of legs the pictured table has. Existing works typically address this problem by utilizing a supervised segmentation model for legs, and then applying a count operator on extracted segmentation masks \citep{Mao2019TheNC}. However, as illustrated in Figure \ref{fig:teaser}, in many visual reasoning tasks, the correct answer depends on a very small portion of a given image, such as a highly occluded back leg, which many existing 2D segmentation systems may neglect.
Humans are able to accurately ground the concepts from images and answer such questions by reasoning on the underlying 3D representation of the image \citep{spelke1993gestalt}. In this underlying 3D representation, the individual legs of a table are roughly similar in size and shape, without underlying issues of occlusion of different legs, making reasoning significantly easier and more flexible. In addition, such an intermediate 3D representation further abstracts confounding factors towards reasoning, such as the underlying viewing direction from which we see the input shape.  To resemble more human-like reasoning capability, in this paper, we propose a novel concept grounding framework by utilizing an intermediate 3D representation.

A natural question arises -- what makes a good intermediate 3D representation for concept grounding? Such an intermediate representation should be discovered in a self-supervised manner and efficient to infer, as well as maintain correspondences between 3D coordinates and feature maps in a fully differentiable and flexible way, so that segmentation and concept learning can directly emerge from this inferred 3D representation with natural supervision of question answering. While traditional 3D representations (\textit{e.g.}, point clouds, voxelgrids and meshes) are efficient to infer, they can not provide
 continuous 3D  feature correspondences flexibly, thus making it challenging to ground concepts to 3D coordinates based on the language descriptions of objects being referred to. To address both issues, in this work, we propose to leverage the continuous, differentiable nature of neural fields as the intermediate 3D representation, which could be easily used for segmentation and concept learning through question answering.

 \begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{fig/teaser6.pdf}

\caption{\small \textbf{Issues with 2D Concept Grounding and Visual Reasoning. } Existing methods typically answer questions by relying on 2D segmentation masks obtained from supervised models. However, heavy occlusion leads to incorrect segmentations and answers.}
\vspace{-10pt}
\label{fig:teaser}

\end{figure}

To parameterize a neural field for reasoning, we utilize recently proposed Neural Descriptor Fields (NDFs) \citep{simeonov2021neural} which assign each 3D point in a scene a high dimensional descriptor. Such descriptors are learned in a self-supervised manner, and implicitly capture the hierarchical nature of a scene, by leveraging implicit feature hierarchy learned by a neural network. Portions of a scene relevant to a particular concept could be then differentiably extracted through a vector similarity computation between each descriptor and a corresponding vector embedding of the concept, enabling concept segmentations on NDFs to be differentiably learned. In contrast, existing reasoning approaches utilize supervised segmentation models to pre-defined concepts, which prevents reasoning on concepts unseen by the segmentation model \citep{Mao2019TheNC}, and further prevents models from adapting segmentations based on language descriptions.

On top of NDFs, we define a set of neural operators for visual reasoning. First, we construct a filter operator, and predict the existence of a particular concept in an image. We also have a query operator which queries about an attribute of the image. In addition, we define a neural counting operator, which quantifies the number of instances of a particular concept. We find that by integrating our neural operators with NDF, our framework is able to effectively and robustly answer a set of visual reasoning questions. Simultaneously, we observe the emergence of natural 3D segmentations of concepts, at the semantic and instance level, directly from the underlying supervision of downstream reasoning.

Our contributions can be summed up as follows: 1) we propose 3D-CG, which utilizes the differentiable nature of neural descriptor fields (NDF) to ground concepts and perform segmentations; 2) we define a set of neural operators, including a neural counting operator on top of the NDF; 3) with 3D-CG, semantic and instance segmentations can naturally emerge from question answering supervision; 4) our 3D-CG outperforms baseline models in both segmentation and reasoning tasks; 5) it can also generalize well to unseen shape categories and real scans.

\section{3D Concept Grouding}
In Figure \ref{fig:framework}, we show an overall framework of our 3D Concept Grounding (3D-CG), which seamlessly bridges the gap between 3D perception and reasoning by using Neural Descriptor Field (NDF). Specifically, we first use NDF to assign a high dimensional descriptor vector for each 3D coordinate, and run a semantic parsing module to translate the question into neural operators to be executed on the neural field. The concepts, also the parameters of the operators, possess concept embeddings as well. Upon executing the operators and grounding concepts in the neural field, we perform dot product attention between the descriptor vector of each coordinate and the concept embedding vector to calculate the score (or mask probability) of a certain concept being grounded on a coordinate. We also propose a count operator which assigns point coordinates into different slots for counting the number of instances of a part category. Both the NDF and the neural operator have a fully differentiable design. Therefore, the entire framework can be end-to-end trained, and the gradient from the question answering loss can be utilized to optimize both the NDF perception module and the visual reasoning module and jointly learn all the embeddings and network parameters.
\begin{figure*}
    \centering
    \includegraphics[width=\linewidth]{fig/framework_final.pdf}

    \caption{\small \textbf{Our 3D Concept Grounding (3D-CG) framework.} Given an input image and a question, 3D-CG first utilizes a Neural Descriptor Field (NDF) to extract a descriptor vector for each 3D point coordinate (here we take 5 coordinates with 5 different colors as examples), and a semantic parsing network to parse the question into operators. The mask probabilities of the coordinates are initialized with occupancy values. Concepts are also mapped to embedding vectors. Attention is calculated between the descriptors and concept embeddings to yield new mask probabilities and filter the set of coordinates being referred as the concept. We also define a count operator which segments a part category into different slots. The sum of the maxpooled mask probabilities of all slots can be used for counting, and the loss can be back propagated to optimize NDF as well as the embeddings. Semantic and instance segmentations naturally emerge with 3D-CG.}
    \label{fig:framework}
    \vspace{-10pt}
\end{figure*}

\subsection{Model Details}
\subsubsection{Neural Descriptor Fields}
In this paper,  we utilize a neural field to map a 3D coordinate $\mathbf{x}$ using a descriptor function $f(\mathbf{x})$ and extract the feature descriptor of that 3D coordinate:
\begin{equation}
f(\mathbf{x}): \mathbb{R}^{3} \rightarrow \mathbb{R}^{n}, \quad \mathbf{x} \mapsto f(\mathbf{x})=\mathbf{v}
\end{equation}
where $\mathbf{v}$ is the descriptor representation which encodes information about shape properties (\textit{e.g.}, geometry, appearance, \textit{etc}).

In our setting, we condition the neural field on the partial point cloud $\mathcal{P}$ derived from RGB-D images. We use a PointNet-based encoder $\mathcal{E}$ to encode $\mathcal{P}$. The descriptor function then becomes $f(\mathbf{x}, \mathcal{E}(P)) = \mathbf{v}$. This continuous and differentiable representation maps each 3D coordinate $\mathbf{x}$ to a descriptor vector $\mathbf{v}$. Since the concepts to be grounded in most reasoning tasks focus on geometry and appearance, we leverage two self-supervised pre-training tasks to parameterize $f$ and learn the features concerning these properties.

\myparagraph{Shape Reconstruction.} Inspired by recent works on using occupancy networks \citep{Mescheder2019OccupancyNL} to reconstruct shapes and learn shape representations, we also use an MLP decoder $\mathcal{D}_1$ which maps each descriptor vector $\mathbf{v}$ to an occupancy value: $\mathcal{D}_1: \mathbf{v}\mapsto \mathbf{o} \in [0,1]$,
where the occupancy value indicates whether the 3D coordinate is at the surface of a shape.

\myparagraph{Color Reconstruction.} We use another MLP $\mathcal{D}_2$ to decode an RGB color of each coordinate: $\mathcal{D}_2: \mathbf{v} \mapsto \mathbf{c} \in \mathbb{R}^{3}$, where the color value indicates the color of the 3D coordinate on the shape.

After learning decoders $D_i$ through these self-supervised pre-training tasks, to construct the Neural Descriptor Field (NDF) $f({\bm{x}})$, we concatenate all the intermediate activations of $D_i$ when decoding a 3D point $\mathbf{x}$ \citep{simeonov2021neural}. The resultant descriptor vector $\mathbf{v}$ can then be used for concept grounding. Since $\mathbf{v}$ consists of intermediate activations of $D_i$, they implicitly capture the hierarchical nature of a scene, with earlier activations corresponding to lower level features and later activations corresponding to higher level features.

\subsubsection{Concept Quantization}
Visual reasoning requires determining the attributes (\textit{e.g.}, color, category) of a shape or shape part, where each attribute contains a set of visual concepts (\textit{e.g.}, blue, leg). As shown in Figure \ref{fig:framework}, visual concepts such as legs, are represented as vectors in the embedding space of part categories. These concept vectors are also learned by our framework. To ground concepts in the neural fields, we calculate the dot products $\langle\cdot, \cdot\rangle$ between the concept vectors and the descriptor vectors from the neural field. For example, to calculate the score of a 3D coordinate belonging to the category \texttt{leg}, we take its descriptor vector $\mathbf{v}$ and compute $p = \langle \mathbf{v}, v^{leg}\rangle$, where $v^{leg}$ is the concept embedding of \texttt{leg}.

\subsubsection{Semantic Parsing}
To transform natural language questions into a set of primitive operators that can be executed on the neural field, a semantic parsing module is incorporated into our framework. This module utilizes a LSTM-based Seq2Seq to translate questions into a set of fundamental operators for reasoning. These operators takes concepts and attributes as their parameters (\textit{e.g.}, \texttt{Filter(leg)}, \texttt{Query(color)}).

\subsubsection{Neural operators on Neural Descriptor Fields}
The operators extracted from natural language questions can then be executed on the neural field. Due to the differentiable nature of the neural field, the whole execution process is also fully-differentiable. We represent the intermediate results in a probabilistic manner: for the $i$-th sampled coordinate $\mathbf{x}_i$, it is represented by a descriptor vector $\mathbf{v}_i$ and has an attention mask $\mathbf{m}_i \in [0,1]$. $\mathbf{m}_i$ denotes the probability that a coordinate belongs to a certain set, and is initialized using the occupancy value output by $\mathcal{D}_1$: $\mathbf{m}_i = \mathbf{o}_i$.

There are three kinds of operators output by the semantic parsing module, we will illustrate how each operator executes on the descriptor vectors to yield the
outputs.

\myparagraph{Filter Operator.} The filter operator ``selects outs" a set of coordinates belonging to the concept $c$ by outputting new mask probabilities:
\vspace{5pt}
\begin{equation}\text{Filter}(c): \mathbf{m}_{i}^c = min(\mathbf{m}_i, \langle \mathbf{v}_i, v^{c}\rangle)\end{equation}
\vspace{-8pt}
\myparagraph{Query Operator.} The query operator asks about an attribute $a$ on selected coordinates and outputs concept $\hat{c}$ of the maximum probability:
\vspace{5pt}
\begin{equation}\text{Query}(a): \hat{c} = \arg \max min(\mathbf{m}_i, \langle \mathbf{v}_i, v^{c}\rangle), c \in a\end{equation}

\myparagraph{Count Operator.} The count operator intends to segment the coordinates in the same part category into instances and count the number of instances. Inspired by previous unsupervised segmentation methods which output different parts in various slots \citep{Locatello2020ObjectCentricLW, Chen2019BAENETBA}, we also allocate a set of slots for different instances of a category. The $j$-th slot $s_j$ has its own embedding vector $v^s_j$. The score $s_{ij}$ of the $i$-th coordinate belonging to the $j$-th slot is also computed by dot product $s_{ij} = \langle \mathbf{v}, v^{s_j}\rangle$. We further take softmax to normalize the probabilities across slots. Since we want to count the instances of a part that was previously selected out, we also take the minimum of previous mask probabilities and the mask probabilities of each slot. The result of counting (\textit{i.e.}, the number of part instances $n$) is obtained by summing up the maximum probability in each slot.

\begin{align}
\text{Count}(c): \mathbf{m}_{ij} &= min(\mathbf{m}_i^c, \frac{s_{ij}}{\sum s_{i'j}})\\
n &= \sum_j \max_i \mathbf{m}_{ij}
\end{align}

\subsubsection{Segmentation}
\label{sec:segmentation}
Note that based on the above operators, not only visual reasoning can be performed, but we can also do semantic segmentation and instance segmentation.

\myparagraph{Semantic Segmentation.} We can achieve semantic segmentations by executing \texttt{Query(category)} on each coordinate and output the category $\hat{c}$ with the maximum probability for each point. \myparagraph{Instance Segmentation.} We can perform instance segmentations by first doing semantic segmentation, and then for each output $\hat{c}$ we further execute \texttt{count}$(\hat{c})$. \\

\subsection{Training Paradigm}
\vspace{1mm}
\myparagraph{Optimization Objective.}
During training, we jointly optimize the NDF and the concept embeddings by minimizing the loss as: \begin{equation}\mathcal{L} = \alpha \cdot \mathcal{L}_{\text {NDF}}+\beta\cdot \mathcal{L}_{\text {reasoning}}\end{equation}

Specifically, the NDF loss consists of two parts: the binary cross entropy classification loss between the ground-truth occupancy and the predicted occupancy, and the MSE loss between the ground-truth rgb value and the predicted rgb value.

We also define three kinds of losses for three types of questions: 1) For questions that ask about whether a part exist, we use an MSE loss $\|a-\max({\mathbf{m}_i})\|^2$ where $a$ is the ground-truth answer and $\max({\mathbf{m}_i})$ takes the maximum mask probability among all sampled 3D coordinates; 2) For questions that query about an attribute, we take the cross entropy classification loss between the predicted concept category $\hat{c}$ and the ground-truth concept category; 3) For counting questions, we first use an MSE loss $\|a-n\|^2$ between the answer and the number output by summing up the maxpool values of all the slots, and further add a loss to ensure that the mask probabilities of the top $K$ values in the top $a$ slots (\textit{i.e.}, the slots with the maximum maxpool values) should be close to 1, where $K$ is a hyper-parameter.
During training, we first train the NDF module only for $N_1$ epochs and then jointly optimize the NDF module and the concept embeddings. The Seq2seq model for semantic parsing is trained independently with supervision.

\myparagraph{Curriculum Learning.} Motivated by previous curriculum strategies for visual reasoning \citep{Mao2019TheNC}, we employ a curriculum learning strategy for training. We split the questions according to the length of operators they are parsed into. Therefore, we start with questions with only one neural operator (\textit{e.g.}, ``is there a yellow part of the chair" can be parsed into \texttt{Filter(Color)}).

\section{Related works}

\myparagraph{Visual Reasoning. } There have been different tasks focusing on learning visual concepts from natural language, such as visually-grounded question answering \citep{gan2017vqs,Ganju2017WhatsIA} and text-image retrieval \citep{Vendrov2016OrderEmbeddingsOI}. Visual reasoning stands out as a challenging task as it requires human-like understanding of the visual scene. Numerous visual reasoning models have been proposed in recent years. Specifically, MAC \citep{Hudson2018CompositionalAN} combined multi-modal attention for compositional reasoning. LCGN \citep{Hu2019LanguageConditionedGN} built contextualized representations for objects to support relational reasoning. These methods model the reasoning process implicitly with neural networks. Neural-symbolic methods \citep{Yi2018NeuralSymbolicVD, Mao2019TheNC,chen2021grounding} explicitly perform symbolic reasoning on the objects representations and language representations. Specifically, they use perception models to extract 2D masks for 3D shapes as a first step, and then execute operators and ground concepts on these pre-segmented masks, but are limited to a set of pre-defined concepts. In this paper, we present an approach towards grounding 3D concepts in a fully differentiable manner, with which 3D segmentations can emerge naturally with question answering supervision.

\myparagraph{Neural Fields.} Our approach utilizes neural fields also known as neural implicit representations, to parameterize an underlying 3D geometry of a shape for reasoning.  Such fields have been shown to accurately represent 3D geometry \citep{Niemeyer2019ICCV,chen2019learning,park2019deepsdf,peng2020convolutional,rebain2021deep}, dynamic scenes \citep{niemeyer2019occupancy,du2021nerflow}, and appearance \citep{sitzmann2019srns,mildenhall2020nerf,Niemeyer2020DVR,yariv2020multiview,saito2019pifu}, acoustics \citep{luo2022learning} and more general multi-modal signals \citep{du2021gem}. In this work, we utilize descriptors defined on such fields \citep{simeonov2021neural} to differentiably reason in 3D.

\myparagraph{Language-driven Segmentation. } Recent works have been focused on leveraging language for segmentation. Specifically, BiLD \citep{Gu2021ZeroShotDV} distills the knowledge from a pre-trained zero-shot image classification model into a two-stage detector. MDETR \citep{Kamath2021MDETRM} is an end-to-end modulated detector that detects objects in an image conditioned on a raw text query, like a caption or a question. LSeg \citep{li2022lseg} uses a text encoder to compute embeddings of descriptive input labels together with a transformer-based image encoder that computes dense per-pixel embeddings of the input image. GroupViT \citep{Xu2022GroupViTSS} learns to group image regions into progressively larger arbitrary-shaped segments with a text encode. These methods typically use an encoder to encode the text and do not have the ability to modify the segmentations based on the question answering loss. LORL \citep{Wang2021LanguageMediatedOR} uses the object-centric concepts derived from language to facilitate the learning of object-centric representations. However, they can only improve the segmentation results but cannot generate segmentations from scratch with language supervision.


\subsection*{Multifarious DNA tile system design.}

Prior theoretical proposals\autocite{Murugan2015-ps,Zhou2015-ym} for multifarious mixtures require each component to accept multiple strongly binding partners at each binding site. However, in DNA tile assembly, each binding site can usually only bind its Watson-Crick complement, and not an arbitrary set of other domains.
Hence, we used an alternate approach: we assumed three structures made of entirely unique tiles and ran a merging algorithm that attempted to reuse tiles in a roughly checkerboard pattern across structures. The algorithm accepted merges if consequences for unintentional binding between other tiles were minimal. After determining the abstracted shared and tile layout in this way, we designed DNA sequences reflecting this layout.

The three target shapes were drawn on a 24 $\times$ 24 single-stranded tile (SST) molecular canvas\autocite{wei_complex_2012}, at an abstract level, with each location in each shape initially a being unique tile with four abstract binding sites referred to as `glues'. (After sequence design, matching glues will correspond to binding pairs of complementary sequence domains.) Edges of the shapes used a special `null glue' that is intended to have no valid binding partner.  In total, this initial design had 2,706 glues, and 1,456 tiles. The three shapes were then processed through a `merging' algorithm that attempted to reuse the same tiles in different shapes.  Randomly choosing two tiles in two different shapes (with null glues on the same sides of each tile, if any), each step of the algorithm attempted to make the two tiles identical by reusing the same four glues in both, and propagating the changes in the glues to all places they occurred within all shapes, starting with the neighboring tiles (e.g., \efigref{extfig:merging}[a]).  To avoid having these changes create undesired growth pathways, for example, allowing chimera of multiple shapes, the modified set was checked after each trial merging step for two criteria from algorithmic self-assembly (\efigref{extfig:merging}[b]).  The self-healing criterion requires that, for any correct subassembly of any shape, while attachments of the wrong tile for a particular location may take place by one bond, only the correct tile can attach by two or more bonds\autocite{winfree_self-healing_2006}.  The second-order sensitivity criterion requires that, for any correct subassembly of any shape, if an incorrect attachment by one bond takes place, the incorrectly attached tile will not create a neighborhood where an additional incorrect tile can attach by two bonds, and thus the initial error will be likely to fall off\autocite{evans_optimizing_2018}.  These two criteria, which are trivially satisfied when every tile and bond is unique to a particular location, continue to be satisfied after each merging step. Thus, we ensure that there is at least a minimum barrier to continued incorrect growth in a regime where tile attachment by two or more bonds is favorable, and attachment by one bond is unfavorable, which is the case close to the melting temperature of most DNA tile assembly systems\autocite{winfree_simulations_1998,evans_physical_2017}.

The algorithm repeatedly merged tiles that satisfied the two criteria until no further acceptable merges were possible.  As each merge could affect the acceptability of later merges by changing the glues around each tile, in order to guide the algorithm toward a sequence of merges more likely to be compatible, the algorithm was initially restricted to considering an alternating `checkerboard' subset of tiles, which, apart from edges, were likely to be merge-able. After exhausting possible merges of these subsets, the algorithm then attempted merges using all tiles in the system.  After repeating this stochastic algorithm multiple times, and selecting the system with the smallest number of tiles, the final resulting system had 698 binding domains and 917 tiles, with 371 of tiles shared between at least two shapes (\efigref{extfig:merging}[c]).

After the assignment of abstract binding domains to each tile by the merging algorithm, the sequences for the binding domains, and thus tiles themselves, were generated using the sequence design software of Woods and Doty et al\autocite{woods_doty_diverse_2019}.  Tiles used a standard SST motif, with alternating 10 and 11 nt binding domains, designed to have similar binding strengths. Following Woods and Doty et al\autocite{woods_doty_diverse_2019}, we set a target range of -8.9 to -9.2~kcal/mol for a single domain at 53\,\ensuremath{^\circ\text{C}}\xspace, which was between the melting temperature and growth temperature for their system.  Null binding domains on the edges of shapes, not intended to bind to any other tiles, were assigned poly-T sequences.

\subsection*{Models of nucleation.}

To model the dependence of the nucleation rates of the three shapes on patterns of unequal concentration, we developed a simple nucleation model based on the stochastic generation of possible nucleation pathways and critical nuclei.  The model estimates nucleation rates by analyzing stochastic paths generated in a greedy manner by making single-tile additions starting from a particular monomer in the system. At each step, all favorable attachments are added and then an unfavorable attachment is performed with probability weighted by the relative free-energy differences of the available tile attachment positions. When multiple favorable attachments are available, the most favorable attachment is made deterministically. This procedure is repeated for many paths over all possible initial positions within the shape considered, and the barrier (highest free energy state visited in `growing' a full structure) is recorded for each path. A nucleation rate is estimated by assuming an equilibrium occupation of this barrier state (Arrenhius' approximation \autocite{frenkel}) and summing over the kinetics of the available attachments from this state. See \efigref{extfig:nucleation} and \sisgm for a detailed discussion.  The approximations here could be improved by running fully reversible simulations, e.g., using Forward Flux Sampling \autocite{Allen2005-ve}.

\subsection*{Fluorophore labels, DNA synthesis, and growth.}

Sites for fluorophore and quencher modifications were chosen to avoid edges, modify only unshared tiles, and provide a reasonable distribution of locations on each shape.  Fluorophores were chosen for spectral compatibility and temperature stability\autocite{you_measuring_2011}.  ROX, ATTO550, and ATTO647N were paired with Iowa Black RQ, and FAM was paired with Iowa Black FQ.  Both fluorophore and quencher modifications were made on the 5' ends of tiles; to sufficiently co-localize fluorophores and quenchers, one tile in the label pair used a reversed orientation (\figref{fig:flagscan}[a]).  Fluorophore labels are discussed in detail in \sifluorophores.

Tiles without fluorophore or quencher modifications were ordered unpurified (desalted) and normalized to 400~\ensuremath{\text{\textmu M}}\xspace in TE buffer (Integrated DNA Technologies).  Strands with fluorophore or quencher modifications were ordered HPLC-purified and normalized to 100~\ensuremath{\text{\textmu M}}\xspace.
Individual tiles were mixed, in the concentration patterns used for experiments, using an Echo 525 acoustic liquid handler (Beckman Coulter). Samples used TEMg buffer (TE buffer with 12.5~mM~MgCl$_2$) in a total volume of $\sim$ 20~\ensuremath{\text{\textmu L}}\xspace.  Flag experiments used a 50~nM base concentration of unenhanced tiles, and an 880~nM concentration of enhanced concentration tiles, while pattern recognition experiments employed tiles with nominal concentrations between 16.67~nM to 450~nM, which were then quantized into ten discrete values to simplify mixing and conserve material (see \sipatmix).  Fluorophore and quencher-modified tile locations always had tiles mixed at the lowest concentration used in the experiment.
Given that unpurified synthetic oligonucleotides typically have less than 40\% to 60\% of the molecules being full length, it is remarkable (though consistent with Woods and Doty et al\autocite{woods_doty_diverse_2019}) that this did not prevent successful pattern recognition by nucleation.

For flag experiments, and pattern recognition of trained images, four samples were prepared per concentration pattern: one sample for each shape with all four fluorophore labels on only that shape, to monitor growth of multiple regions on each shape, and an additional sample with one fluorophore on each shape: ROX, ATTO550 (`five'), and ATTO647N (`six') on H, A, and M respectively.  For pattern recognition of test images, for experiment size reasons, only the lattermost sample was prepared.\setbox0\vbox{\autocite{xgrow}\autocite{schulman_programmable_2009}}

After transferring 20~\ensuremath{\text{\textmu L}}\xspace to PCR tubes, samples were grown in an mx3005p quantitative PCR (qPCR) machine (Agilent), in order to provide a program of controlled temperature over time while monitoring fluorescence.  Growth protocols began with a ramp from 71\,\ensuremath{^\circ\text{C}}\xspace to 53\,\ensuremath{^\circ\text{C}}\xspace over 40 minutes to ensure any potentially preexisting complexes were melted, and then a slower ramp from 53\,\ensuremath{^\circ\text{C}}\xspace to an initial growth temperature at 1\,\ensuremath{^\circ\text{C}}\xspace per hour.  At this point, three different protocols were used.  For constant temperature flag growth experiments, the growth temperature was 47\,\ensuremath{^\circ\text{C}}\xspace, and this was held for 51 hours.  For temperature ramp flag growth, the initial growth temperature was 48\,\ensuremath{^\circ\text{C}}\xspace, which was reduced over 100 hours to 46\,\ensuremath{^\circ\text{C}}\xspace.  For pattern recognition, a ramp from 48\,\ensuremath{^\circ\text{C}}\xspace to 45\,\ensuremath{^\circ\text{C}}\xspace over 150 hours was used.  For constant temperature experiments, fluorescence readings were taken every 12 minutes, and for other experiments, every 30 minutes.  After the growth period, temperature was lowered to 39\,\ensuremath{^\circ\text{C}}\xspace at 1\,\ensuremath{^\circ\text{C}}\xspace per 26 minutes.  See \sitprots for temperature protocols plotted as a function of time. Because of the small sample size and long experiment duration, great care to avoid evaporation was necessary.  Once protocols were finished, samples were stored at room temperature until ready for AFM imaging.

AFM imaging was performed using a FastScan AFM (Bruker) in fluid tapping mode.  To achieve better images, two techniques were combined: sample warming to prevent nonspecific clumping of structures, and washing with Na-supplemented buffer to prevent smaller material, such as unbound, single DNA tile strands, from adhering to the mica surface.  Each sample was diluted 50x into TEMg buffer with an added 100~mM~NaCl, then warmed to approximately 40\,\ensuremath{^\circ\text{C}}\xspace for 15 minutes.  50~\ensuremath{\text{\textmu L}}\xspace of the sample mix was deposited on freshly-cleaved mica, then left for two minutes.  As much liquid as possible was pipetted off of the mica and discarded, then immediately replaced with Na-supplemented buffer again, and mixed by pipetting up and down.  This washing process of buffer removal and addition was repeated twice with added-Na buffer, then once with TEMg buffer to remove remaining Na, before imaging was performed in TEMg buffer.  As adhesion of DNA to mica is dependent upon the ratio of monovalent and divalent cations in the imaging buffer, this process was meant to ensure that unbound tiles were removed during the washing process where Na and Mg were present, while imaging itself took place with only Mg, so that the lattice structures would be more strongly adhered to the surface, resulting in better image quality.

\subsection*{Fluorescence and AFM data analysis.}

Fluorophore signals are known to be affected by extraneous factors such as temperature, pH, secondary structure, and the local base sequence near the fluorophore\autocite{you_measuring_2011}, which complicates quantitative interpretation of absolute fluorescence levels.
Our own control experiments also illustrated effects due to partial assembly intermediates as well as due to the total amount of single stranded DNA in solution (\sifluorophores).
For this reason, the fluorescence of each fluorophore was normalized to the maximum raw fluorescence value of that fluorophore in that particular sample, and the \emph{time} at which the fluorescence signal decreased by 10\% was then
used as a measure of the extent of nucleation that appears less sensitive to these artifacts (\efigref{extfig:fluorophores}).
The duration between the point of 10\% quenching and the end of the growth segment of the experiment was defined as the `growth time' for that fluorophore label; the growth time was defined as 0 in the event of quenching never reaching 10\%.  For concentration patterns with four samples with different fluorophore arrangements, the total growth time of a shape was defined as the average of the growth time of the five total fluorophore labels on the shape across the four samples (four in the shape-specific sample, and one in the each-shape sample), while for concentration patterns with only one sample, the growth time of the corresponding fluorophore label was used.  As the position of the fluorophore within the shape, relative to where nucleation occurs, has a substantial influence on growth time measurements, the considerable variability in these measurements relative to the true nucleation kinetics must be acknowledged.

For flag experiments, AFM imaging was done only for qualitative confirmation of the selective nucleation and growth indicated by fluorescence results.  For pattern recognition and equal-concentration experiments, however, shapes in AFM images were uniformly quantified.  At least one sample of each of the patterns had three 5 $\times$ 5 $\mum$  images taken under comparable conditions.  The sample corresponding with each image was blinded, and structures were counted independently by each of the four authors, classifying structures as either ``nearly complete'' or ``clearly identifiable'' examples of each of the three shapes.  For the purposes of analysing pattern-dependent nucleation and growth, no clear distinction between the number of nearly complete and clearly identifiable shapes was found, and so the two categories were summed.  Counts were averaged across the three images, then averaged across the counts of the four authors, to obtain a count per shape per 25 $\mum^2$ region for each pattern.
Each author used their own, subjective, interpretation of ``nearly complete'' and ``clearly identifiable'' structures, and the total number of structures counted in each image differed by up to $\pm 50\%$ for different authors.  However, the ratios of different shapes in each image counted by each author remained within 5\% of the mean ratios for most images, and across all images, no author had a bias of more than $\pm 4\%$ toward identifying a particular shape more or less often than average.

To measure the selectivity of patterns, the fraction of on-target shape growth time, and AFM counts, compared to the sum of shape growth times and AFM counts, was used.  The total growth times, and total AFM counts, of the on-target shapes were used to measure overall shape growth.

\subsection*{Pattern recognition training.}

Images for pattern recognition were selected from several sources, rescaled to 30 $\times$ 30, discretized to 10 grayscale values, and adjusted so that the number of pixels with each value was consistent across all images (see \siimages for details).  Each pixel's grayscale value, $0 \le p_n \le 1$, was converted to the concentration $c_i$ for the corresponding tile $t_i$ using an exponential formula, $c_i = c e^{3\;p_n\ln 3}$, where the base concentration is $c=16.67$~nM. The intention of the numbers used was to make the average tile concentration 60~nM for each image.  As each image had 900 pixels and there are 917 tiles in the system, 17 tiles did not have their concentrations set by any pixel; these tile concentrations were uniformly set to the lowest concentration, and the assignment of these tiles was used to ensure that fluorophore label locations did not vary in concentration.

The tile-pixel assignment was optimized through a simple hill-climbing algorithm, starting from a random assignment, where random modifications to the assignment map are attempted at each step and accepted if the move increases the efficacy of the map. This efficacy was quantified through a heuristic function that accounts for relative nucleation rates, location of nucleation sites (with emphasis given to locations that succeeded in the flag experiments shown in \figref{fig:flagscan}[d]), and satisfaction of constraints related to the fluorescent reporters. Because the nucleation algorithm described above is costly, a simplistic model of nucleation based upon the Boltzmann-weighted sum of concentrations over a $k \times k$ window swept over each structure (similar to the model employed in Zhong et al\autocite{zhong_associative_2017}) was used to evaluate relative nucleation rates for a majority of the optimization steps. The more detailed but computationally costly model described above was then employed for an additional several hours in hopes of improving the mapping.  The window-based nucleation model (along with all constraints about nucleation location and fluorescent reporters) is also employed to explore the capacity of this map training procedure in \efigref{extfig:capacity}.  Details of the pattern recognition training and the window-based nucleation model are discussed in \sialltraining.

\printbibliography[segment=2,check=onlynew]

\end{methods}

\begin{addendum}
	\item[Acknowledgements.]
	We thank Michael Brenner, Jehoshua Bruck, Aaron Dinner, David Doty, Deborah K. Fygenson,  Stanislas Leibler, Richard M. Murray, Lulu Qian, Paul W. K. Rothemund, Petr \v{S}ulc, Chris Thachuk, Grigory Tikhomirov, Damien Woods, and Zorana Zeravcic.  Ting Zhu, Thomas Ouldridge, Salvador Buse, Matthew Alexander, Mohini Misra, Anna Lapteva also provided valuable feedback on early drafts.
   \item[Funding.]
	   Supported by National Science Foundation grants CCF-1317694 and CCF/FET-2008589,
	   the Evans Foundation for Molecular Medicine,
	   European Research Council grant 772766, Science Foundation Ireland grant 18/ERCS/5746,
	   and the Carver Mead New Adventures Fund.
JOB, AM were primarily supported by the University of Chicago
Materials Research Science and Engineering Center, which is funded by National
Science Foundation under award number DMR-2011854. AM acknowledges support from the Simons Foundation.
   \item[Author Contributions.]
	   CGE, EW, AM conceived the study.
	   CGE and EW designed the molecules.
	   CGE, JOB, EW, AM wrote simulation code, designed the experiments and performed the experiments, analyzed the data and wrote the manuscript.
   \item[Competing Interests.]
	   The authors declare that they have no competing financial interests.
   \item[Correspondence.]
	   Correspondence should be addressed to \texttt{cge@dna.caltech.edu},
	   \texttt{jdobrien07@gmail.com},
	   \texttt{winfree@caltech.edu}, or \texttt{amurugan@uchicago.edu}.
   \end{addendum}

\renewcommand\thefigure{E\arabic{figure}}
\setcounter{figure}{0}

\begin{figure*}[ht]
\centering
\includegraphics[width=6.1in]{Figures/extfig0-hopfield-vs-sham.pdf}
\caption{\textbf{Parallels and differences between neural network models and self-assembly models as exemplars of collective behavior.}
In this rough metaphor, a neuron corresponds to a tile.  While Hopfield networks allow full connectivity, multifarious self-assembly (like place cell networks) restricts connectivity to a superposition of grids with different unit permutations.  The state of a Hopfield network consists of the set of active neurons, while the state of an assembly consists of the set of tiles present and their arrangement, which is restricted to be connected.
The energy of a state is a quadratic function governed by synaptic weights $w_{i,j}$ and biases $b_i$ for neural activities , or for assemblies, by directional binding energies $J^\delta_{i,j}$ for tiles $i$ and $j$ at positions $p$ and $p'$ that are neighbors in direction $\delta$, along with tile chemical potentials $\Theta_i$.
An environment presents a sequence of outside influences driving system state.
Learning in Hopfield networks occurs any time neurons are simultaneously active; for self-assembly, they must also be located next to each other.
Qualitative system behaviors depend on the number of memories being stored and the operating temperature, including phases where system state randomizes (paramagnetic / disoriented / dissolved), gets locked in a spurious local minimum (spin glass / random aggregation), or successfully retrieves learned memories.
Due in large part to the restrictions on connectivity, the capacity of place cell networks and multifarious self-assembly is less than for the Hopfield model.
See \sineural for details and discussion.}
\label{extfig:neural}
\end{figure*}

\begin{figure*}[ht]
\centering
\includegraphics{Figures/extfig1-tileassignment.pdf}
\caption{\textbf{Proofreading tile set design and tile assignment map.}
\textbf{a,} Our systems are designed to grow in a regime where a tile attaching by at least two bonds is favorable, but a tile attaching by one bond is not (`threshold 2').   Motivated by self-healing tile systems~\autocite{winfree_self-healing_2006}, we seek a tile set where no correct partial assembly should ever allow an undesired tile to attach by two or more bonds, though undesired attachments by one bond are allowed, such that any favorable attachment to a partial assembly will be correct.
\textbf{b,} In addition to tiles attaching favorably by $2$ bonds to growing facets, new facets in the system will only be created by tiles attaching unfavorably by one bond, and then being stabilized by further, favorable growth.
At a site where tile $T$ would correctly attach by one bond, a tile $U$ might be able to attach incorrectly by the same bond.  $T$ would correctly be stabilized by the subsequent attachment of $V$ by two bonds, but $U$ might be able to be stabilized as well if there is a tile $W$ that can attach to it and shares the same glue as $V$.
Thus, if for every pair of tiles that can bind to each other (e.g., $T+V$), there is no other pair of binding tiles (e.g., $U+W$) that share two glues on the same edges of the tiles, then any tile that attaches by one bond to an assembly will either be the correct tile, or will not allow a subsequent attachment, and will likely detach.
This is equivalent to `second-order sensitivity' with all directions treated as inputs, functioning as a form of `proofreading'~\autocite{evans_optimizing_2018,winfree_proofreading_2004}.
\textbf{c,} We created a multifarious tile system by first starting with three shapes constructed entirely of unique tiles, then repeatedly attempting to `merge' tiles in different shapes by constraining the sequences of their domains to be identical, and checking whether each merge of two tiles results in a tile system that does not have any tile pairs violating criteria in \textbf{a} and \textbf{b}. \textbf{d,}
From multiple trials of the merging process, each initially favoring a checkerboard arrangement before attempting more general merges, we selected the smallest result containing 917 tiles.
DNA sequences for tiles in the system were designed with the single-stranded tile (SST) motif\autocite{yin_programming_2008}, with two alternating tiles motifs of 11~nt and 12~nt domains (tile sequences and full shape layouts are shown in \sitiles).}

\label{extfig:merging}
\end{figure*}

\begin{figure*}[ht]
\centering
\includegraphics{Figures/ExtFig2-proofreading-mod-alpha.pdf}
\caption{\textbf{Suppression of chimeric growth through tile set design.}
\textbf{a,} We use simulations to contrast assembly errors in three distinct tile sets: the proofreading tile set with an inert boundary used in experiments, described in \figref{fig:multifarious} \textbf{(a, top)}; %
a simple checkerboard tile set with a strictly alternating shared and unique tile pattern for each shape, where %
unique tiles can be seen as mediating different interactions between shared tiles \textbf{(a, middle)}; and an edge-guarded checkerboard in which we additionally enforce inert bonds around each shape's perimeter \textbf{(a, bottom)}.
For each tile set, we performed kinetic growth simulations, starting from a pre-formed $5\times 5$ seed taken from a location within H.
Simulations were performed using the kinetic Tile Assembly Model as implemented by XGrow (with chunk fission) \autocite{xgrow} with uniform tile concentrations corresponding to 62 nM and parameters estimated in \siktam.
\textbf{b,} Schematic illustrates various desired and undesired growth pathways for A, along with representative AFM images taken from the A flag 1 experiment
(\siAflag1).  Two distinct kinds of chimeric structures were seen in simulation as the result of promiscuous interactions: chimeric structures can grow either before full assembly of the target structure (e.g., part-A, part-M) or emerge spontaneously from the edge of a properly formed structure (e.g. full-A, part-H). Chimeras like those illustrated along the lower path are held together by just a few bonds and sometimes can quickly break apart (tiles with unintended bonds are shown in red); these result in sharp drops in simulated assembly size, as the simulation discards one subassembly when disconnected.
Note that chimeric growth was not observed experimentally, possibly as a result of effective experimental system design; however, many observed structures failed to complete the upper right and/or lower left corners, or appeared to have suffered a spiral growth defect.
A possible explanation for the missing corners, which is also seen in H and M, is supported by coarse-grained molecular dynamics simulations of SST lattice curvature (\sioxDNA).  Spiral defects were not seen in H or M and are presumably due to the interior hole in A.
\textbf{c--e,} The size of the assembly (in units of the size of the fully formed H) is shown as a function of time.
For higher temperature 48.9\,\ensuremath{^\circ\text{C}}\xspace (\textbf{c}), no chimeras are observed on the simulated timescales for any tile set. For intermediate temperature 47.2~\ensuremath{^\circ\text{C}}\xspace (\textbf{d}), all 6 checkerboard trajectories still result in chimeras, while no errors are observed on the timescale probed for the guarded checkerboard or experimentally-implemented proofreading tile set.
For lower temperature 45.5\,\ensuremath{^\circ\text{C}}\xspace (\textbf{e}), chimeras are seen in all runs for checkerboard structures (red traces), 4 of the 6 runs for guarded checkerboard structures (green traces) and 1 of the 6 runs for proofreading structures.
}
  \label{extfig:errorsims}%
\end{figure*}

\begin{figure*}
    \centering
    \includegraphics{Figures/extfig4-nucleation.pdf}
    \caption{%
\textbf{Stochastic Greedy Nucleation Model, based on repeated stochastic simulations.}
\textbf{a,} The frequently-used kinetic Tile Assembly Model (kTAM)~\autocite{winfree_simulations_1998,evans_physical_2017} has rates for tile attachment and detachment events based on tile and assembly diffusion and total binding strength of correct attachments a tile can make at a lattice site.  Here $u_0 = 1$~M.
\textbf{b,} These rates can be used to derive a free energy for any tile assembly in a system, and, assuming fixed monomer concentrations, an equilibrium concentration for any assembly. Schulman \& Winfree \autocite{schulman_programmable_2009} showed that the equilibrium concentration of the highest-energy assembly along a nucleation trajectory under this assumption provides an upper bound for nucleation rate through that trajectory, with or without fixed monomer concentrations.
However, in a large system, considering all possible intermediate assemblies and all pathways, including many that are extremely unlikely, would be infeasible.
Thus, we developed the Stochastic Greedy Nucleation Model to generate stochastically-chosen paths of tile attachments.
\textbf{c,} Starting from a single tile (chosen with probability proportional to relative concentration), whenever the assembly is in a state $A_\mathrm{stable}$ where there is no tile attachment that would be favorable (have $\Delta G < 0$), one of the possible unfavorable (with $\Delta G \geq 0$) attachments is stochastically chosen, resulting in a higher-$G$ state $A_\mathrm{unstable}$. Then, all subsequent possible $\Delta G < 0$ attachments are made, resulting in the next $A'_\mathrm{stable}$ state; for our system of unique tiles for each site in the lattice, this sequence of favorable steps has a unique resulting assembly.
\textbf{d,} The process repeats until all tiles in a shape are attached, which results in a trajectory with a maximum-$G$ assembly that can be used to bound the rate of nucleation, $\eta^+$, through that particular trajectory.
\textbf{e,} By using this process to collect many trajectories, and then repeating the entire process for each of the three shapes in the system, we can estimate nucleation rates dependent upon temperature, with the assumption that tile monomer concentrations do not deplete, and that the trajectories found are a reasonable representation of likely trajectories.
For comparison between model predictions and experimental data in \efigref{extfig:flags}[d] and \ref{extfig:patterns}b,
we determined the temperature at which the model predicted the nucleation rate exceeded a threshold (orange line), to compare with when fluorescence quenching exceeded a threshold.
For details on the SGM model, see \sisgm.
\textbf{f,} To study the winner-take-all effect, we use a simplfied CRN model (shown here) and a more detailed model based on (but simpler than) the SGM; see \siwtacrn.
}
\label{extfig:nucleation}
\end{figure*}

\begin{figure*}
    \centering
    \includegraphics{Figures/extfig3-labels.pdf}
    \caption{\textbf{Fluorophore quenching as a measure of nucleation and growth.}
    \textbf{a,} Fluorescent labels used a fluorophore-quencher pair placed on the $5'$ ends of two modified tiles unique to one shape, where they were co-located, but had no complementary binding domains, ensuring that dimers could not form, and trimers would not closely co-locate the fluorophore and quencher.  To constrain the pair to be close enough to quench in a well-formed lattice, one of the two tiles had its orientation and crossover position swapped compared to the unmodified tile for the location.
    \textbf{b,} Positions and types of all fluorophore/quencher pairs available for use.  For one sample, one position for each of four types of fluorophores could be chosen, and tile pairs for those locations replaced by their modified counterparts.  Thus different samples could probe different arrangements of up to four locations; four arrangements were used in experiments (e.g., in \textbf{e}).
    \textbf{c,} Expected behavior of fluorophore labels on shapes as one shape nucleates and grows.
    \textbf{d,} Fluorescence data for non-quenching (fluorophore tile only, orange) and quenching ($5\times 5$ lattice around fluorophore and quencher tiles, blue) controls for the ATTO647N fluorophore/quencher pair on A. Here, the temperature ramps linearly from $49\,\ensuremath{^\circ\text{C}}\xspace$ to $35\,\ensuremath{^\circ\text{C}}\xspace$  at a rate of $0.1\,\ensuremath{^\circ\text{C}}\xspace/\text{min}$, with all tiles at 50~nM, and each sample has its fluorescence normalized to its maximum value independently.
    \textbf{e,} An example of fluorescence growth time measurements (Mockingbird; see \simockingbird).
    Each fluorophore signal, in each sample, is independently normalized to its maximum value during the experiment, and the time between the point where the signal goes below 0.9 (``10\% quenching'') and the end of the experiment is measured (``growth time'').
    These times are then summed for all fluorophores, in all four samples, on each shape, resulting in a growth time for each shape, and, when normalized to the sum of all growth times, a relative growth time for each shape.
    See Methods and \sifluorophores\ for design and characterization of the fluorescence readout method, as well as an estimate of the melting temperature of H.
    }
    \label{extfig:fluorophores}
\end{figure*}

\begin{figure*}
    \centering
    \includegraphics{Figures/extfig5-flags.pdf}
    \caption{\textbf{Nucleation and growth with `flag' patterns of enhanced concentration.}
    \textbf{a--c,} 36 different concentration patterns with enhanced concentrations of shared tiles in $5\times 5$ regions were prepared, each with four different standard sets of fluorophores in four samples, and grown using two temperature protocols (\textbf{a}): a ramp focusing on $48\,\ensuremath{^\circ\text{C}}\xspace$ to $46\,\ensuremath{^\circ\text{C}}\xspace$ over 100 hours, and a hold at $47\,\ensuremath{^\circ\text{C}}\xspace$.
    Using growth times as described in \efigref{extfig:fluorophores}, fluorescence data for many samples in both experiments showed preference for the desired shapes (\textbf{b, c}), but with considerable variation in selectivity and total amount of growth.  %
    \textbf{d,} No statistically significant correlation was found between the nucleation model prediction for temperature of on-target nucleation and the time of on-target shape quenching in the temperature ramp experiment.
    \textbf{e--g,} Details of three patterns, with concentration patterns (1), weighted critical nucleus free energy starting from particular tiles (2), nucleation-model-estimated nucleation rates (3), temperature hold (4) and temperature ramp (5) experiment fluorescence results, and (6) AFM images from the temperature hold experiments. Information for all individual flag patterns is available in the \siflagdata.    }
    \label{extfig:flags}
\end{figure*}

\begin{figure*}[ht]
\begin{center}
\includegraphics{Figures/ExtFig6_expandWTAPanel-redone.pdf}
\caption{\textbf{Evidence of winner-take-all in flag experiments.} \textbf{a,} An example flag pattern (A flag 12), and uniform 50 nM concentration `no flag' pattern. \textbf{b,} Fluorescence normalized to maximum readings, from the temperature ramp experiment (\efigref{extfig:flags} and \siflagdata).  The fluorescence at the end of the experiment, $F_{p,s,f}$, of fluorophore $f$ in sample $s$ of pattern $p$ is used along with the value for the corresponding fluorescence value for the no flag pattern, $F_{\mathsf{no\, flag},s,f}$, to calculate the ratio $M_{p,s,f}$.  This ratio corresponds to the relative amount of quenching for that fluorophore in the flag pattern compared to the no flag pattern.  The ratios are averaged across the 5 on-target fluorophores (circled in \textbf{b}) in samples for the flag pattern to obtain an average on-target ratio, and across the 10 off-target fluorophores to obtain an average off-target ratio.  \textbf{c,} The on- and off-target ratios are plotted for each flag pattern.  For winner take all behavior, on-target quenching is expected to be higher with a flag pattern than with no flag, resulting in $F_p^{\mathsf{on\,target}} > 1$, while off-target quenching is expected to be reduced, resulting in $F_p^{\mathsf{off\,target}} < 1$. \label{extfig:wta}}

\end{center}
\end{figure*}

\begin{figure*}
    \centering
    \includegraphics{Figures/ExtFig9-capacity-mod.pdf}
    \caption{\textbf{Pattern recognition capacity.} To analyze the pattern-recognition capabilities of the designed tile set, the map-training algorithm (see \simaptraining) was run for increasingly larger sets of random images.
    \textbf{a-c,} Example images mapped to concentration patterns for sets with 1, 12, and 18 trained images per shape, with the intended target shape for each image indicated.  Following the same procedure as used for the experimental system, with the same weighting of locations, $30\times 30$ images with 10 possible grayscale values and matching histograms were mapped exponentially to tile concentrations in the 917 tile system; however, all images were generated randomly.  Training was done using only the Window Nucleation Model with a window size $k$ of either 2, 4, or 6, with a limit of 400,000 steps (\siwindow).  For each number of images per shape considered, ten repetitions of training (starting from random assignments) were performed (to account for variability of the training algorithm) for each of three different sets of images (to account for variability in sets of images).
    \textbf{d,} As the number of images in the set increases, the performance of the training algorithm decreases. For larger $k$, the pixel-tile map can exploit higher-order correlations and can thus accommodate more images.  For each system, nucleation rates were calculated using the Stochastic Greedy Model, described in \sisgm, at $\ensuremath{G_{\mathrm{se}}}=5.4$, which roughly corresponds to a temperature of 48.6\,\ensuremath{^\circ\text{C}}\xspace, and with concentrations comparable to the experimental system.  Selectivity was calculated as the nucleation rate of the target shape for each image divided by the total nucleation rate of all three shapes for that image, averaged over all images in the system, and over all 30 systems (10 repetitions for each of 3 sets of images) for each point, with 90\% confidence intervals shown.  Star shows specificity calculated from nucleation model results for the experimentally-implemented system.
    \textbf{e,} As the number of images increases, pattern recognition must increasingly rely on patterns of concentrations of shared tiles, rather than choosing a pixel-to-tile map that places high-concentration pixels on tiles unique to the target shape.  Histograms show average concentrations of tiles in different shapes or combinations of shape
    (including the average across tile categories)
    for images in training cases \textbf{a--c}, and the experimental system.  The change can also be seen in the concentration maps of \textbf{a--c}, with the sharp checkerboard of high concentration tiles in target shapes in \textbf{a} becoming less apparent in \textbf{b} and \textbf{c}.
    }
    \label{extfig:capacity}
\end{figure*}

\begin{figure*}
    \centering
    \includegraphics{Figures/extfig8-patterns.pdf}
    \caption{\textbf{Classification of images viewed as concentration patterns.}  36 different concentration patterns, derived from a mapping of 36 grayscale images, were run using a ramp between 48\,\ensuremath{^\circ\text{C}}\xspace to 45\,\ensuremath{^\circ\text{C}}\xspace over approximately 150 hours.
    \textbf{a,} Three pattern examples, with source image, concentration pattern, nucleation model nucleation rate starting from particular tiles, nucleation model nucleation rates, fluorescence results, and AFM images. \textbf{b,} Across all patterns there was some correlation between the on-target nucleation temperature predicted by the nucleation model and on-target shape quenching time. \textbf{c,} Total AFM shape counts for each sample.  Information for all patterns is available in the \sipatterndata.} %
    \label{extfig:patterns}
\end{figure*}

\end{document}