You will be shown sections from a scientific research paper, together with a question about the paper. Your task is to answer the question based on the contents of the paper. Your response should be a `json` object with two fields:

"answer": An array of strings extracted directly from the paper which, collectively, answer the question.
"evidence": An array of strings. Each should be an excerpt from the paper, in which one or more of the extracted answers can be found.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": ["BERT", "RoBERTa"],
  "evidence": ["In our experiments, we compare the performance of our model against BERT and RoBERTa."]
}

Do not include any text in your response other than the json.

If the question is unanswerable given the provided excerpts, respond with the single word "null".

Paper: Understanding Car-Speak: Replacing Humans in Dealerships

A large portion of the car-buying experience in the United States involves interactions at a car dealership. At the dealership, the car-buyer relays their needs to a sales representative. However, most car-buyers are only have an abstract description of the vehicle they need. Therefore, they are only able to describe their ideal car in "car-speak". Car-speak is abstract language that pertains to a car's physical attributes. In this paper, we define car-speak. We also aim to curate a reasonable data set of car-speak language. Finally, we train several classifiers in order to classify car-speak.

Introduction
A large portion of the car-buying experience in the United States involves interactions at a car dealership BIBREF0, BIBREF1, BIBREF2. Traditionally, a car dealer listens and understands the needs of the client and helps them find what car is right based on their needs.
With the advent of the internet, many potential car buyers take to the web to research cars before going to a dealership in person BIBREF0, BIBREF2. However, nearly 50% of customers bought a car at the dealership based on the sales representative's advice, not their own research BIBREF1, BIBREF2.
Throughout this interaction the dealer is acting as a type of translator or classifier. The dealer takes a natural language input (e.g. “I need a fast, family friendly, reliable car under $20k”) and returns a list of suggestions. The dealer understands the ideas of “fast”, “family friendly”, and “reliable” and is able to come up with a reasonable recommendation based on this knowledge.
In this paper we aim to create a system that can understand car-speak based on some natural language input (we want to recreate the dealer from above). But how do we prepare a proper training set for a Natural Language model? What model is best suited to this problem? Can this model take a human out of the car-buying process? To answer these questions, the remainder of this paper makes the following contributions:
Defining “car-speak” and its role in the car-buying process.
Appropriate training data for a Natural Language model.
A model that is able to properly classify car-speak and return a car.
We aim to accomplish these goals in a scientific manner, using real data and modern methods.

Related Work
There has been some work done in the field of car-sales and dealer interactions. However, this is the first work that specifically focuses on the
Deloitte has published a report on the entire car-buying process BIBREF0. The report goes into great depth about the methods potential buyers use to find new cars to buy, and how they go about buying them. The report tells us that there are several unique phases that a potential buyer goes through before buying a car.
Verhoef et al. looked at the specifics of dealer interaction and how dealers retain customers BIBREF3. Verhoef tells us how important dealers are to the car-buying process. He also explains how influential a dealer can be on what car the buyer purchases.
Jeff Kershner compiled a series of statistics about Dealership Sales BIBREF1. These statistics focus on small social interactions BIBREF4 between the dealer and the buyer.
Barley explains the increasing role of technology in the car-buying process BIBREF2. Barley tells us that users prefer to use technology/robots to find the cars they want to buy instead of going to a dealer, due the distrust towards sales representatives.

What is Car-speak?
When a potential buyer begins to identify their next car-purchase they begin with identifying their needs. These needs often come in the form of an abstract situation, for instance, “I need a car that goes really fast”. This could mean that they need a car with a V8 engine type or a car that has 500 horsepower, but the buyer does not know that, all they know is that they need a “fast” car.
The term “fast” is car-speak. Car-speak is abstract language that pertains to a car's physical attribute(s). In this instance the physical attributes that the term “fast” pertains to could be the horsepower, or it could be the car's form factor (how the car looks). However, we do not know exactly which attributes the term “fast” refers to.
The use of car-speak is present throughout the car-buying process. It begins in the Research Phase where buyers identify their needs BIBREF0. When the buyer goes to a dealership to buy a car, they communicate with the dealer in similar car-speak BIBREF2 and convey their needs to the sales representative. Finally, the sales representative uses their internal classifier to translate this car-speak into actual physical attributes (e.g. `fast' $ \longrightarrow $ `700 horsepower & a sleek form factor') and offers a car to the buyer.
Understanding car-speak is not a trivial task. Figure FIGREF4 shows two cars that have high top speeds, however both cars may not be considered “fast”. We need to mine the ideas that people have about cars in order to determine which cars are “fast” and which cars are not.

Gathering Car-speak Data
We aim to curate a data set of car-speak in order to train a model properly. However, there are a few challenges that present themselves: What is a good source of car-speak? How can we acquire the data? How can we be sure the data set is relevant?
What is a good source of car-speak? We find plenty of car-speak in car reviews. Table TABREF5 provides excerpts from reviews with the car-speak terms bolded. Car reviews often describe cars in an abstract manner, which makes the review more useful for car-buyers. The reviews are often also about specific use-cases for each car (e.g. using the car to tow a trailer), so they capture all possible aspects of a car. The reviews are each written about a specific car, so we are able to map car-speak to a specific car model.
We choose the reviews from the U.S. News & World Report because they have easily accessible full-length reviews about every car that has been sold in the United States since 2006 BIBREF5.
How can we acquire the data? We can acquire this data using modern web-scraping tools such as beautiful-soup. The data is publicly available on https://cars.usnews.com/cars-trucks BIBREF5. These reviews also include a scorecard and justification of their reviews.
How can we be sure the data set is relevant? On average vehicles on United States roads are 11.6 years old, making the average manufacturing year 2006-2007 BIBREF6, BIBREF7. In order to have a relevant data set we gather all of the available reviews for car models made between the years 2000 and 2018.

Translating Car-Speak
Our data set contains $3,209$ reviews about 553 different cars from 49 different car manufacturers. In order to accomplish our goal of translating and classifying car-speak we need to filter our data set so that we only have the most relevant terms. We then need to be able to weight each word in each review, so that we can determine the most relevant ideas in each document for the purpose of classification. Finally, we need to train various classification models and evaluate them.

Translating Car-Speak ::: Filtering the Data
We would like to be able to represent each car with the most relevant car-speak terms. We can do this by filtering each review using the NLTK library BIBREF8, only retaining the most relevant words. First we token-ize each review and then keep only the nouns and adjectives from each review since they are the most salient parts of speech BIBREF9. This leaves us with $10,867$ words across all reviews. Figure FIGREF6 shows the frequency of the top 20 words that remain.
Words such as “saftey” and “luxury” are among the top words used in reviews. These words are very good examples of car-speak. Both words are abstract descriptions of cars, but both have physical characteristics that are associated with them as we discussed in Section SECREF3.

Translating Car-Speak ::: TF-IDF
So far we have compiled the most relevant terms in from the reviews. We now need to weight these terms for each review, so that we know the car-speak terms are most associated with a car. Using TF-IDF (Term Frequency-Inverse Document Frequency) has been used as a reliable metric for finding the relevant terms in a document BIBREF10.
We represent each review as a vector of TF-IDF scores for each word in the review. The length of this vector is $10,867$. We label each review vector with the car it reviews. We ignore the year of the car being reviewed and focus specifically on the model (i.e Acura ILX, not 2013 Acura ILX). This is because there a single model of car generally retains the same characteristics over time BIBREF11, BIBREF12.

Translating Car-Speak ::: Classification Experiments
We train a series of classifiers in order to classify car-speak. We train three classifiers on the review vectors that we prepared in Section SECREF8. The classifiers we use are K Nearest Neighbors (KNN), Random Forest (RF), Support Vector Machine (SVM), and Multi-layer Perceptron (MLP) BIBREF13.
In order to evaluate our classifiers, we perform 4-fold cross validation on a shuffled data set. Table TABREF10 shows the F1 micro and F1 macro scores for all the classifiers. The KNN classifier seem to perform the best across all four metrics. This is probably due to the multi-class nature of the data set.

Conclusion & Future Work
In this paper we aim to provide an introductory understanding of car-speak and a way to automate car dealers at dealerships. We first provide a definition of “car-speak” in Section SECREF3. We explore what constitutes car-speak and how to identify car-speak.
We also gather a data set of car-speak to use for exploration and training purposes. This data set id full of vehicle reviews from U.S. News BIBREF5. These reviews provide a reasonable set of car-speak data that we can study.
Finally, we create and test several classifiers that are trained on the data we gathered. While these classifiers did not perform particularly well, they provide a good starting point for future work on this subject.
In the future we plan to use more complex models to attempt to understand car-speak. We also would like to test our classifiers on user-provided natural language queries. This would be a more practical evaluation of our classification. It would also satisfy the need for a computer system that understands car-speak.

Figure 1: Both of these cars can achieve high speeds. Which is “fast”?

Table 1: Excerpts from car reviews.

Figure 2: The frequencies of the top 20 words in reviews.

Table 2: Evaluation metrics for all classifiers.

Question: What is the performance of classifiers?
{"answer": ["Table TABREF10", " The KNN classifier seem to perform the best across all four metrics. This is probably due to the multi-class nature of the data set", " While these classifiers did not perform particularly well, they provide a good starting point for future work on this subject"], "evidence": ["In order to evaluate our classifiers, we perform 4-fold cross validation on a shuffled data set. Table TABREF10 shows the F1 micro and F1 macro scores for all the classifiers. The KNN classifier seem to perform the best across all four metrics. This is probably due to the multi-class nature of the data set."]}