text:
\section{Introduction}

Modern Internet service providers pervasively employ AI to augment their services, as the advancement of AI technology has brought breakthroughs in processing images, video, speech, and audio ~\cite{lecun2015deep},
and hence boost the deployments of massive AI algorithms, systems and architectures. For example, Alibaba proposes a new DUPN network for more effective personalization~\cite{ni2018perceive}. Facebook integrates AI into many essential products and services like news feed~\cite{hazelwood2018applied}. Google proposes the TensorFlow~\cite{abadi2016tensorflow} system and the tensor processing unit (TPU)~\cite{jouppi2017datacenter} to accelerate the service performance. Amazon adopts AI for intelligent product recommendation~\cite{smith2017two}.

Consequently, the pressures of measuring and evaluating these algorithms, systems, and architectures rise.
First, the real world data sets and workloads from Internet services are treated as first-class  confidential issues by their providers, and  they  are isolated between academia and industry, or even among different  providers. However,  there are only a few publicly available performance model or observed insights~\cite{hazelwood2018applied,ayers2018memory} about industry-scale Internet services that can be leveraged for further research.
As there is no publicly available industry-scale Internet service benchmark, the state-of-the-art and state-of-the-practice are advanced only by the research staffs inside Internet service providers, which is not sustainable and poses a huge obstacle for our communities towards developing an open and mature research field.

Second, AI has infiltrated into almost all aspects of Internet services, ranging from offline analytics to online service. Thus,
to cover the critical paths and characterize prominent characteristics of a realistic AI scenairo, end-to-end application benchmarks should be provided~\cite{barroso2009datacenter,bigbench}.
Meanwhile, there are many classes of Internet services. Modern Internet services workloads expand and change very fast, and it is not scalable or even impossible
to create a new benchmark or proxy for every possible workload~\cite{gao2018motif}.  Moreover, data sets have great impacts on system and microarchitectural characteristics~\cite{xie2018cvr,gao2018motif}, so diverse data inputs should be considered. So we need identify representative data sets, abstract the prominent AI problem domains (component benchmarks), and further understand what are the most intensive  units of computation (micro benchmarks), on the basis of which, we can build a concise and comprehensive AI benchmark framework.

Finally but not least, from an architectural perspective, porting a full-scale AI applications to a new architecture at an earlier stage is difficult and even impossible~\cite{bailey1991parallel}, while using micro or component benchmarks alone are insufficient to discover the time breakdown of different modules and locate the bottleneck within a realistic AI application scenario at a later stage~\cite{bailey1991parallel}.
Hence, a realistic  AI benchmark suite should have the ability to run not only collectively as a whole end-to-end application to discover the  time breakdown of different modules but also individually as a micro or component benchmark for fine tuning hot spot functions or kernels.
The state-of-the-art or state-of-the-practise ~\cite{mlperf,adolf2016fathom,deepbench,dong2017dnnmark,
coleman2017dawnbench} AI benchmarks only provide a few micro or component benchmarks, and none of them is able to cover the full use cases  of an  industry-scale  Internet service. So an industry standard Internet service AI benchmark suite consisting of a full spectrum of micro or component benchmarks and an end-to-end application benchmark is of great significance to bridge this huge gap.

To the best of our knowledge, this paper presents the first industry scale AI benchmark suite, AIBench, joint with seventeen industry partners.
First, we present a highly extensible, configurable, and flexible benchmark framework, containing multiple loosely coupled modules like data input, prominent AI problem domains, online inference, offline training and automatic deployment tool modules. We analyze  typical AI application scenarios from three most important Internet services domains, including search engine, social network, and e-commerce, and then we abstract and identify sixteen prominent  AI problem domains, including \emph{ classification, image generation, text-to-text translation, image-to-text, image-to-image, speech-to-text, face embedding, 3D face recognition, object detection, video prediction, image compression, recommendation, 3D object reconstruction, text summarization, spatial transformer, and learning to rank}. We implement sixteen component benchmarks for those AI problem domains, and further profile and implement  twelve  fundamental units of computation across different component benchmarks as the micro benchmarks.
On the basis of the AIBench framework, we design and implement the first end-to-end Internet service AI benchmark with an underlying e-commerce searching business model. As a whole, it covers the major modules and critical paths of an industry scale e-commerce provider. The application benchmark reuses ten component benchmarks from the AIBench framework, receives the query requests and performs
personalized searching, recommendation and advertising, integrated with AI inference and training.
The data maintains the real-world data characteristics through anonymization. Data generators are also provided to generate specified data scale, using several configurable parameters.

In summary, our contributions are four-fold as follows.

\begin{itemize}

\item We propose and implement a highly extensible, configurable, and flexible AI benchmark framework.
\item We identify sixteen prominent AI problem domains with seventeen industry partners, and implement sixteen component benchmarks targeting those domains accordingly.
\item We design and implement the first industry scale end-to-end  Internet service AI  benchmark suite, with an underlying e-commerce searching model.

\item On the CPU and GPU clusters, we perform workload characterizations of the end-to-end Internet service AI benchmark. We found that AI-related components significantly change the critical paths of the online service and deteriorate the average and tail latency, and the architects must perform a serious trade-off between the service quality and the complexity of neural network model. For offline training, we identify six kernels, consistent with our micro benchmarks~\footnote{The six kernels are named based on CUDA functions and they  are a subset of our micro benchmarks.}, and the corresponding function calls that takes up the most percentages of  running time, and analyze the GPU execution stalls that influence the performance, which provide the guide for further optimizations.

\end{itemize}

The rest of this paper is organized as follows. In Section 2, we present the related work. Section 3 summarizes the AIBench framework. Section 4 illustrates the design and implementation of an end-to-end application benchmark. Section 5 illustrates the experiment configurations. In Section 6, we present the characterization results on GPUs and CPUs.
Finally, we draw the conclusion in Section 7.

\section{Related Work}

AI attracts great attention, appealing many research efforts on benchmarks. Table~\ref{comparition_table} compares AIBench  with respect to the state-of-the-art or state-of-the-practice AI benchmark suites, from the perspectives of end-to-end application benchmarks, component benchmarks, micro benchmarks, real-world data sets and software stacks.

\begin{table}[htbp]
\renewcommand\arraystretch{1.2}
\scriptsize
\centering
\caption{AI Benchmark Comparison.}\label{comparition_table}
\center
\begin{tabular}{|p{0.8in}|p{0.18in}|p{0.4in}|p{0.4in}|p{0.4in}|p{0.5in}|p{0.5in}|p{0.55in}|p{0.4in}|}
\hline
\multicolumn{2}{|c|}{} & \tabincell{c}{AIBench} &\tabincell{c}{MLPerf} &\tabincell{c}{Fathom}
&\tabincell{c}{DeepBench}
&\tabincell{c}{DNNMark}
&\tabincell{c}{DAWNBench}
&\tabincell{c}{TBD}\\
\hline
\rowcolor{mygray} \multicolumn{9}{|l|}{End-to-End Application Benchmark}\\
\hline
\multicolumn{2}{|c|}{Online module} & \CheckmarkBold & $\times$ & $\times$ & $\times$ & $\times$ & $\times$ & $\times$\\
\hline
\multicolumn{2}{|c|}{Offline module} & \CheckmarkBold & $\times$ & $\times$ & $\times$ & $\times$ & $\times$ & $\times$\\
\hline
\rowcolor{mygray} \multicolumn{9}{|l|}{Component Benchmark}\\
\hline
\multirow{2}{*}{\tabincell{l}{Image\\classification}} & \tabincell{l}{Train} & \CheckmarkBold & \CheckmarkBold & \CheckmarkBold & $\times$ & $\times$ & \CheckmarkBold & \CheckmarkBold \\
\cline{2-9}
& \tabincell{l}{Infer} & \CheckmarkBold & \CheckmarkBold & \CheckmarkBold & $\times$ & $\times$ & \CheckmarkBold &  $\times$ \\
\hline
\multirow{2}{*}{\tabincell{l}{Image\\generation}} & \tabincell{l}{Train} & \CheckmarkBold & $\times$ & $\times$ & $\times$ & $\times$ & $\times$ & \CheckmarkBold \\
\cline{2-9}
& \tabincell{l}{Infer} & \CheckmarkBold & $\times$ & $\times$ & $\times$ & $\times$ & $\times$ & $\times$\\
\hline
\multirow{2}{*}{\tabincell{l}{Text-to-Text}} & \tabincell{l}{Train} & \CheckmarkBold & \CheckmarkBold & \CheckmarkBold & $\times$ & $\times$ & $\times$ & \CheckmarkBold\\
\cline{2-9}
& \tabincell{l}{Infer} & \CheckmarkBold & \CheckmarkBold & \CheckmarkBold & $\times$ & $\times$ & $\times$ & $\times$\\
\hline
\multirow{2}{*}{\tabincell{l}{Image-to-Text}} & \tabincell{l}{Train} & \CheckmarkBold & $\times$ & $\times$ & $\times$ & $\times$ & $\times$ & $\times$\\
\cline{2-9}
& \tabincell{l}{Infer} & \CheckmarkBold & $\times$ & $\times$ & $\times$ & $\times$ & $\times$ & $\times$\\
\hline
\multirow{2}{*}{\tabincell{l}{Image-to-Image}} & \tabincell{l}{Train} & \CheckmarkBold & $\times$ & $\times$ & $\times$ & $\times$ & $\times$ & $\times$\\
\cline{2-9}
& \tabincell{l}{Infer} & \CheckmarkBold & $\times$ & $\times$ & $\times$ & $\times$ & $\times$ & $\times$\\
\hline
\multirow{2}{*}{\tabincell{l}{Speech recog-\\nition}} & \tabincell{l}{Train} & \CheckmarkBold & \CheckmarkBold & \CheckmarkBold & $\times$ & $\times$ & $\times$ & \CheckmarkBold\\
\cline{2-9}
& \tabincell{l}{Infer} & \CheckmarkBold & \CheckmarkBold & \CheckmarkBold & $\times$ & $\times$ & $\times$ & \CheckmarkBold\\
\hline
\multirow{2}{*}{\tabincell{l}{Face \\embedding}} & \tabincell{l}{Train} & \CheckmarkBold & $\times$ & $\times$ & $\times$ & $\times$ & $\times$ & $\times$\\
\cline{2-9}
& \tabincell{l}{Infer} & \CheckmarkBold & $\times$ & $\times$ & $\times$ & $\times$ & $\times$ & $\times$\\
\hline
\multirow{2}{*}{\tabincell{l}{3D Face \\Recognition}} & \tabincell{l}{Train} & \CheckmarkBold & $\times$ & $\times$ & $\times$ & $\times$ & $\times$ & $\times$\\
\cline{2-9}
& \tabincell{l}{Infer} & \CheckmarkBold & $\times$ & $\times$ & $\times$ & $\times$ & $\times$& $\times$\\
\hline
\multirow{2}{*}{\tabincell{l}{Object \\detection}} & \tabincell{l}{Train} & \CheckmarkBold & \CheckmarkBold & $\times$ & $\times$ & $\times$ & $\times$ & \CheckmarkBold \\
\cline{2-9}
& \tabincell{l}{Infer} & \CheckmarkBold & \CheckmarkBold & $\times$ & $\times$ & $\times$ & $\times$ & $\times$\\
\hline
\multirow{2}{*}{\tabincell{l}{Recommendation}} & \tabincell{l}{Train} & \CheckmarkBold & \CheckmarkBold & $\times$ & $\times$ & $\times$ & $\times$ & \CheckmarkBold\\
\cline{2-9}
& \tabincell{l}{Infer} & \CheckmarkBold & $\times$ & $\times$ & $\times$ & $\times$ & $\times$& $\times$\\
\hline
\multirow{2}{*}{\tabincell{l}{Video \\prediction}} & \tabincell{l}{Train} & \CheckmarkBold & $\times$ & $\times$ &$\times$ & $\times$ & $\times$ & $\times$\\
\cline{2-9}
& \tabincell{l}{Infer} & \CheckmarkBold & $\times$ & $\times$ & $\times$ & $\times$ & $\times$ &$\times$\\
\hline
\multirow{2}{*}{\tabincell{l}{Image \\compression}} & \tabincell{l}{Train} & \CheckmarkBold & $\times$ & \CheckmarkBold & $\times$ & $\times$ & $\times$&$\times$ \\
\cline{2-9}
& \tabincell{l}{Infer} & \CheckmarkBold & $\times$ & \CheckmarkBold & $\times$ & $\times$ & $\times$ &$\times$\\
\hline
\multirow{2}{*}{\tabincell{l}{3D object re-\\construction}} & \tabincell{l}{Train} & \CheckmarkBold & $\times$ & $\times$ & $\times$ & $\times$ & $\times$ &$\times$\\
\cline{2-9}
& \tabincell{l}{Infer} & \CheckmarkBold & $\times$ & $\times$ & $\times$ & $\times$ & $\times$ &$\times$\\
\hline
\multirow{2}{*}{\tabincell{l}{Text sum-\\marization}} & \tabincell{l}{Train} & \CheckmarkBold & $\times$ & $\times$ & $\times$ & $\times$ & $\times$ &$\times$\\
\cline{2-9}
& \tabincell{l}{Infer} & \CheckmarkBold & $\times$ & $\times$ & $\times$ & $\times$ & $\times$ &$\times$\\
\hline
\multirow{2}{*}{\tabincell{l}{Spatial\\ transformer}} & \tabincell{l}{Train} & \CheckmarkBold & $\times$ & $\times$ & $\times$ & $\times$ & $\times$ &$\times$\\
\cline{2-9}
& \tabincell{l}{Infer} & \CheckmarkBold & $\times$ & $\times$ & $\times$ & $\times$ & $\times$ &$\times$\\
\hline
\multirow{2}{*}{\tabincell{l}{Learning to rank}} & \tabincell{l}{Train} & \CheckmarkBold & $\times$ & $\times$ & $\times$ & $\times$ & $\times$ &$\times$\\
\cline{2-9}
& \tabincell{l}{Infer} & \CheckmarkBold & $\times$ & $\times$ & $\times$ & $\times$ & $\times$& $\times$\\
\hline
\multirow{2}{*}{\tabincell{l}{Games}} & \tabincell{l}{Train} & $\times$ & \CheckmarkBold & \CheckmarkBold & $\times$ & $\times$ & $\times$ &\CheckmarkBold\\
\cline{2-9}
& \tabincell{l}{Infer} & $\times$ & $\times$ & \CheckmarkBold & $\times$ & $\times$ & $\times$ &$\times$\\
\hline
\multirow{2}{*}{\tabincell{l}{Memory network}} & \tabincell{l}{Train} & $\times$ & $\times$ & \CheckmarkBold & $\times$ & $\times$ & $\times$ & $\times$\\
\cline{2-9}
& \tabincell{l}{Infer} & $\times$ & $\times$ & \CheckmarkBold & $\times$ & $\times$ & $\times$ &$\times$\\
\hline
\multirow{2}{*}{\tabincell{l}{Question\\ answering}} & \tabincell{l}{Train} & $\times$ & $\times$ & $\times$ & $\times$ & $\times$ & \CheckmarkBold &$\times$\\
\cline{2-9}
& \tabincell{l}{Infer} & $\times$ & $\times$ & $\times$ & $\times$ & $\times$ & \CheckmarkBold &$\times$\\
\hline
\rowcolor{mygray} \multicolumn{9}{|l|}{Micro Benchmark}\\
\hline
\multicolumn{2}{|c|}{Convolution} & \CheckmarkBold & $\times$ & $\times$ & \CheckmarkBold & \CheckmarkBold & $\times$ & $\times$\\
\hline
\multicolumn{2}{|c|}{Fully connected} & \CheckmarkBold & $\times$ & $\times$ & \CheckmarkBold & \CheckmarkBold & $\times$ &$\times$\\
\hline
\multicolumn{2}{|c|}{\tabincell{c}{Element-wise op}} & \CheckmarkBold & $\times$ & $\times$ & $\times$ & $\times$ & $\times$ &$\times$\\
\hline
\multicolumn{2}{|c|}{Activation} & \CheckmarkBold & $\times$ & $\times$ & $\times$ & \CheckmarkBold & $\times$ &$\times$\\
\hline
\multicolumn{2}{|c|}{Pooling} & \CheckmarkBold & $\times$ & $\times$ & $\times$ & \CheckmarkBold & $\times$ &$\times$\\
\hline
\multicolumn{2}{|c|}{Normalization} & \CheckmarkBold & $\times$ & $\times$ & $\times$ & \CheckmarkBold & $\times$ &$\times$\\
\hline
\multicolumn{2}{|c|}{Dropout} & \CheckmarkBold & $\times$ & $\times$ & $\times$ & \CheckmarkBold & $\times$ &$\times$\\
\hline
\multicolumn{2}{|c|}{Softmax} & \CheckmarkBold & $\times$ & $\times$ & $\times$ & \CheckmarkBold & $\times$ &$\times$\\
\hline
\multicolumn{2}{|c|}{AllReduce} & $\times$ & $\times$ & $\times$ & \CheckmarkBold & $\times$ & $\times$ &$\times$\\
\hline
\rowcolor{mygray} \multicolumn{9}{|l|}{Real-world Data sets and Software Stack}\\
\hline
\multicolumn{2}{|c|}{Text data} & 3 & 1 & 2 & N/A & N/A & 1 & 1\\
\hline
\multicolumn{2}{|c|}{Image data} & 8 & 2 & 2 & N/A & N/A & 2& 4 \\
\hline
\multicolumn{2}{|c|}{3D data} & 2 & 0 & 0 & N/A & N/A & 0 & 0\\
\hline
\multicolumn{2}{|c|}{Audio data} & 1 & 0 & 1 & N/A & N/A & 0 & 2\\
\hline
\multicolumn{2}{|c|}{Video data} & 1 & 0 & 1 & N/A & N/A & 0 & 0\\
\hline
\multicolumn{2}{|c|}{Software Stack} & 3 & 2 & 1 & 1 & 1 & 2 & 4\\
\hline

\end{tabular}
\end{table}

MLPerf~\cite{mlperf} is an ML benchmark suite targeting six AI problem domains, including image classification, object detection, translation, recommendation, speech recognition, and reinforcement learning. For several problem domains, it provides both light-weight and heavy-weight implementations.  Totally, it includes seven benchmarks for training and five benchmarks for inference.

Fathom~\cite{adolf2016fathom} provides eight deep learning component benchmarks implemented with TensorFlow, among which, three of them use different models to solve image classification problem. The Autoenc workload provides a variational autoencoder and can be used to reduce the dimension and compress images. This benchmark suite also lacks of the micro and application benchmarks.

DeepBench~\cite{deepbench} consists of four operations involved in training deep neural networks, including three basic operations and recurrent layer types. Although the recurrent layer is the combination of several basic operations like convolution, while it is still simpler comparing to our component benchmarks.
In total,  this benchmark suite only provides micro benchmarks, and lacks of the component and application benchmarks.

DNNMark~\cite{dong2017dnnmark} is a GPU benchmark suite that consists of a collection of deep neural network primitives. It provides eight micro benchmarks while lacking of component and application benchmarks.

DAWNBench~\cite{coleman2017dawnbench} is a benchmark and competition focusing on end-to-end performance, which means the training or inference time to achieve a state-of-the-art accuracy. It only focuses on two component benchmarks including image classification on CIFAR10 and ImageNet, and question answering on SQuAD, and lacks of the micro and application benchmarks.

TBD Suite~\cite{zhu2018tbd} is a benchmark suite for DNN training. It provides eight neural network models that covers six application domains. Analogously, it only contains component-level benchmarks, and lacks of an end-to-end application benchmark that can depict the entire execution paths of industry scale application.

Additionally, for machine learning and deep learning evaluation, MLModelScope~\cite{dakkak2019model} proposes a specification for repeatable model evaluation and a runtime to measure experiments.

In conclusion, the state-of-the-art and state-of-the-practise AI benchmarks only provide a few micro or component benchmarks, and none of them is
able to characterize an industry-scale Internet service.  AIBench provides a benchmark framework that all benchmarks collectively constitute an end-to-end application benchmark with an underlying industry scale Internet service model, while each individually forms a  micro or component benchmark that supports fine-grained benchmarking for AI systems, architectures, and algorithms.
Meanwhile, the HPC AI benchmarks~\cite{jiang2018hpc}, IoT AI benchmarks~\cite{luo2018iot}, Edge AI benchmarks~\cite{hao2018edge}, the previous version of AIBench for datacenter~\cite{aibench}, and big data benchmarks~\cite{gao2018bigdatabench,wang2014bigdatabench,jia2013characterizing} are also released on the BenchCouncil web site.

\section{AIBench Framework}\label{framework}

In this section, we introduce the AIBench framework from the perspectives of framework architecture, metrics, prominent AI problem domains we identified, implemented micro and component benchmarks, and framework scalability on large-scale clusters.

\subsection{Framework Architecture}

\begin{figure}[tb]
\centering
\includegraphics*[scale=1]{arch-design.pdf}
\caption{AIBench Framework.}
\label{arch-design}
\end{figure}

The AIBench framework provides an universal AI benchmark framework that is flexible and configurable, which is shown in Fig.~\ref{arch-design}.
It provides loosely coupled modules that can be easily configured and extended to compose an end-to-end application, including the data input, AI problem domain, online inference, offline training, and deployment tool modules.

The data input module is responsible to feed data into the other modules. It collects representative real-world data sets from  not only  the authoritative public websites but also our industry partners after anonymization.  The data schema is designed to maintain the real-world data characteristics, so as to alleviate the confidential issue. Based on the data schema, a series of data generators are further provided to support large-scale data generation, like the user or product information. To  cover a wide spectrum of data characteristics, we take diverse data types, e.g., structured, semi-structured, un-structured, and different data sources, e.g., table, graph, text, image, audio, video, into account. Our framework integrates various open-source data storage systems, and supports large-scale data generation and deployment~\cite{ming2014bdgs}.

To achieve diversity and representativeness of our framework, we first identify prominent  AI problem domains that play important roles in most important Internet services domains. And then we provide the concrete implementation of the AI algorithms targeting those AI problem domains as component benchmarks.
Also, we profile the most intensive units of computation across those component benchmarks, and implement them  as a set of micro benchmarks. Both micro and component benchmarks are implemented with the concern for composability, each of which can run collectively and individually.

The offline training and online inference modules are provided to construct an end-to-end application benchmark. First, the offline training module chooses one or more component benchmarks from the AI problem domain module, through specifying the required benchmark ID, input data, and execution parameters like batch size. Then the offline training module trains a model and provides the trained model to the online inference module. The online inference module loads the trained model onto the serving system, i.e., TensorFlow serving. Collaborating with the other non AI-related modules in the critical paths, an end-to-end application benchmark is built.

To be easily deployed on a large-scale cluster, the framework provides deployment tools that contain two automated deployment templates using Ansible and Kubernetes, respectively. Among them, the Ansible templates support scalable deployment on physical machines or virtual machines, while the kubernetes templates are used to deploy on container clusters. A configuration file needs to be specified for installation and deployment, including module parameters like the chosen benchmark ID, input data, and the cluster parameters like nodes, memory, and network information.

\subsection{The Prominent AI Problem Domains}\label{identify}

To cover a wide spectrum of prominent AI problem domains among Internet services, we thoroughly analyze the core scenarios among three primary Internet services, including search engine, social network, and e-commerce, as shown in Table~\ref{problemdomain}. In total, we identify  sixteen representative AI problem domains as follows.

\begin{table*}[htbp]
\caption{Prominent AI Problem Domains in Internet Service.}
\renewcommand\arraystretch{1.05}
\scriptsize
\label{problemdomain}
\center
\begin{tabular}{|p{0.85in}|p{2.4in}|p{2.6in}|}
\hline
\textbf{Internet Service} & \textbf{Core Scenario} & \textbf{Involved AI Problem Domain} \\
\hline
\multirow{9}{*}{Search Engine} & \multirow{2}{*}{Content-based image retrieval (e.g., face, scene)} & Object detection; Classification; Spatial transformer; Face embedding; 3D face recognition \\
\cline{2-3}
& Advertising and recommendation & Recommendation \\
\cline{2-3}
& \multirow{2}{*}{Maps search and translation} & 3D object reconstruction; Text-to-Text translation; Speech recognition \\
\cline{2-3}
& Data annotation and caption (e.g., text, image) & Text summarization; Image-to-Text \\
\cline{2-3}
& Search result ranking & Learning to rank \\
\cline{2-3}
& Image resolution enhancement & Image generation; Image-to-Image \\
\cline{2-3}
& Data storage space and transfer optimization & Image compression; Video prediction \\
\hline

\multirow{9}{*}{Social Network} & Friend or community recommendation & Recommendation; Face embedding; 3D face recognition; \\
\cline{2-3}
& Vertical search (e.g., image, people) & Classification; Spatial transformer;  Object detection;  \\
\cline{2-3}
& Language translation & Text-to-Text translation \\
\cline{2-3}
& Automated data annotation and caption & Text summarization; Image-to-Text; Speech recognition \\
\cline{2-3}
& Anomaly detection (e.g., spam image detection) & Classification \\
\cline{2-3}
& Image resolution enhancement & Image generation; Image-to-Image \\
\cline{2-3}
& Photogrammetry (3D scanning) & 3D object reconstruction \\
\cline{2-3}
& Data storage space and transfer optimization & Image compression; Video prediction \\
\cline{2-3}
& News feed ranking & Learning to rank \\
\hline

\multirow{9}{*}{E-commerce} & Product searching & Classification; Spatial transformer;  Object detection  \\
\cline{2-3}
& Product recommendation and advertising & Recommendation \\
\cline{2-3}
& Language and dialogue translation & Text-to-Text translation; Speech recognition \\
\cline{2-3}
& Automated data annotation and caption & Text summarization; Image-to-Text  \\
\cline{2-3}
& \multirow{2}{*}{Virtual reality (e.g., virtual fitting)} & 3D object reconstruction; Image generation; Image-to-Image \\
\cline{2-3}
& Data storage space and transfer optimization & Image compression; Video prediction \\
\cline{2-3}
& Product ranking & Learning to rank \\
\cline{2-3}
& Facial authentication and payment & Face embedding; 3D face recognition; \\
\hline

\end{tabular}
\end{table*}

\textbf{Classification.} This problem domain is to extract different thematic classes within the input data like an image or a text file, which is a supervised learning problem to define a set of target classes and train a model to recognize. It is a typical task in Internet services or other application domains, and widely used in multiple scenarios, like category prediction and spam detection.

\textbf{Image Generation.} This problem domain aims to provide an unsupervised learning problem to mimic the distribution of data and generate images. The typical scenario of this task includes image resolution enhancement, which can be used to generate high-resolution image.

\textbf{Text-to-Text Translation.} This problem domain need to translate text from one language to another, which is the most important field of computational linguistics. It can be used to translate the search query intelligently and translate dialogue.

\textbf{Image-to-Text.} This problem domain is to generate the description of an image automatically. It can be used to generate image caption and recognize optical character within an image.

\textbf{Image-to-Image.} This problem domain is to convert an image from one representation of an image to another representation. It can be used to synthesize the images with different facial ages and simulate virtual makeup. Face aging can help search the facial images ranging different age stages.

\textbf{Speech recognition.} This problem domain is to recognize and translate the spoken language to text. This task is beneficial for voice search and voice dialogue translation.

\textbf{Face embedding.} This problem domain is to transform a facial image to a vector in embedding space. The typical scenarios of this task are facial similarity analysis and face recognition.

\textbf{3D face recognition.} This problem domain is to recognize the 3D facial information from multiple images from different angles. This task mainly focuses on three-dimensional images and is beneficial to the facial similarity and facial authentication scenario.

\textbf{Object detection.} This problem domain is to detect the objects within an image. The typical scenarios are vertical search like contented-based image retrieval and video object detection.

\textbf{Recommendation.} This problem domain is to provide recommendations. This task is widely used for advertise recommendation, community recommendation, or product recommendation.

\textbf{Video prediction.} This problem domain is to predict the future video frames through predicting previous frames transformation. The typical scenarios are video compression and video encoding, for efficient video storage and transmission.

\textbf{Image compression.} This problem domain is to compress the images and reduce the redundancy~\cite{toderici2017full}. The task is important for Internet service in terms of data storage overhead and data transmission efficiency.

\textbf{3D object reconstruction.} This problem domain is to predict and reconstruct 3D objects~\cite{yan2016perspective}. The typical scenarios are maps search, light field rendering and virtual reality.

\textbf{Text summarization.} This problem domain is to generate the text  summary, which is important for search results preview, headline generation, and keyword discovery.

\textbf{Spatial transformer.} This problem domain is to perform spatial transformations~\cite{jaderberg2015spatial}. An typical scenario of this task is space invariance image retrieval, so that the image can be retrieved even if the image is extremely stretched.

\textbf{Learning to rank.} This problem domain is to learn the attributes of searched content and rank the scores for the results, which is the key for searching service.

\subsection{Micro and Component Benchmarks}

Targeting  the major AI problem domains abstracted in Section~\ref{identify}, we provide the concrete implementation of the AI algorithms. Individually, these algorithm implementations form a series of micro and component benchmarks for fine-grained evaluation. Table~\ref{AIBench_component} and Table~\ref{AIBench_micro} list the component and micro benchmarks in AIBench. In total, AIBench includes sixteen component benchmarks for AI problem domains and twelve micro benchmarks that extract unit of computation from the typical AI algorithms~\cite{gao2018motif}.

\begin{table*}[htbp]
\scriptsize
\caption{Component Benchmarks in AIBench.}
\renewcommand\arraystretch{1.4}
\label{AIBench_component}
\center
\begin{tabular}{|p{0.65in}|p{1.36in}|p{2in}|p{1.6in}|}
\hline
\textbf{No.} & \textbf{Component Benchmark} & \textbf{Algorithm} & \textbf{Data Set}\\
\hline
DC-AI-C1 & Image classification & ResNet50~\cite{he2016deep} &  ImageNet \\
\hline
DC-AI-C2 & Image generation & WassersteinGAN~\cite{arjovsky2017wasserstein} &  LSUN \\
\hline
DC-AI-C3 & Text-to-Text translation & Transformer~\cite{vaswani2017attention} &  WMT English-German \\
\hline
DC-AI-C4 & Image-to-Text & Neural Image Caption Model~\cite{vinyals2017show} &  Microsoft COCO \\
\hline
DC-AI-C5 & Image-to-Image & CycleGAN~\cite{zhu2017unpaired} &  Cityscapes \\
\hline
DC-AI-C6 & Speech recognition & DeepSpeech2~\cite{amodei2016deep} &  Librispeech \\
\hline
DC-AI-C7 & Face embedding & Facenet~\cite{schroff2015facenet} &  LFW, VGGFace2 \\
\hline
DC-AI-C8 & 3D Face Recognition & 3D face models~\cite{vieriu2015facial} &  77,715 samples from 253 face IDs\\
\hline
DC-AI-C9 & Object detection & Faster R-CNN~\cite{ren2015faster} &  Microsoft COCO \\
\hline
DC-AI-C10 & Recommendation & Neural collaborative filtering~\cite{he2017neural} &  MovieLens \\
\hline
DC-AI-C11 & Video prediction & Motion-Focused predictive models~\cite{finn2016unsupervised} &  Robot pushing data set \\
\hline
DC-AI-C12 & Image compression & Recurrent neural network~\cite{toderici2017full} &  ImageNet \\
\hline
DC-AI-C13 & 3D object reconstruction & Convolutional encoder-decoder network~\cite{yan2016perspective} &  ShapeNet Data set\\
\hline
DC-AI-C14 & Text summarization & Sequence-to-sequence model~\cite{nallapati2016abstractive} &  Gigaword data set \\
\hline
DC-AI-C15 & Spatial transformer & Spatial transformer networks~\cite{jaderberg2015spatial} &  MNIST \\
\hline
DC-AI-C16 & Learning to rank & Ranking distillation~\cite{tang2018ranking} &  Gowalla \\
\hline

\end{tabular}
\end{table*}

\begin{table}[htbp]
\caption{Micro Benchmarks in AIBench.}
\renewcommand\arraystretch{1.2}
\scriptsize
\label{AIBench_micro}
\center
\begin{tabular}{|p{1in}|p{2in}|p{2in}|}
\hline
\textbf{No.} & \textbf{Micro Benchmark} & \textbf{Data Set} \\
\hline
DC-AI-M1 & Convolution  & ImageNet, Cifar \\
\hline
DC-AI-M2 & Fully Connected & ImageNet, Cifar \\
\hline
DC-AI-M3 & Relu  & ImageNet, Cifar \\
\hline
DC-AI-M4 & Sigmoid  & ImageNet, Cifar \\
\hline
DC-AI-M5 & Tanh  & ImageNet, Cifar \\
\hline
DC-AI-M6 & MaxPooling   & ImageNet, Cifar \\
\hline
DC-AI-M7 & AvgPooling & ImageNet, Cifar \\
\hline
DC-AI-M8 & CosineNorm~\cite{luo2017cosine}  &  ImageNet, Cifar\\
\hline
DC-AI-M9 & BatchNorm~\cite{ioffe2015batch}   &  ImageNet, Cifar \\
\hline
DC-AI-M10 & Dropout~\cite{srivastava2014dropout}  &  ImageNet, Cifar \\
\hline
DC-AI-M11 & Element-wise multiply  &  ImageNet, Cifar \\
\hline
DC-AI-M12 & Softmax  &  ImageNet, Cifar \\
\hline

\end{tabular}
\end{table}

\subsection{Data Model}

To cover a diversity of data sets from various applications, we collect 15 representative data sets, including ImageNet~\cite{deng2009imagenet}, Cifar~\cite{krizhevsky2014cifar}, LSUN~\cite{yu2015lsun}, WMT English-German~\cite{wmt}, Cityscapes~\cite{cordts2016cityscapes}, LibriSpeech~\cite{panayotov2015librispeech}, Microsoft COCO data set~\cite{lin2014microsoft}, LFW~\cite{huang2008labeled}, VGGFace2~\cite{cao2018vggface2}, Robot pushing data set~\cite{finn2016unsupervised},  MovieLens data set~\cite{harper2016movielens}, ShapeNet data set~\cite{chang2015shapenet}, Gigaword data set~\cite{rush2017neural}, MNIST data set~\cite{lecun2010mnist}, Gowalla data set~\cite{cho2011friendship} and the 3D face recognition data set from our industry partner.

\subsection{Metrics}

AIBench focuses on a series of metrics covering accuracy, performance, and energy consumption, which are major industry concerns.

The metrics for online inference contains query response latency, tail latency, and throughput from performance aspect, inference accuracy, and inference energy consumption.

The metrics for offline training contains the samples processed per second, the wall clock time to train the specific epochs, the wall clock time to train a model achieving a target accuracy~\cite{coleman2017dawnbench}, and the energy consumption to train a model achieving a target accuracy~\cite{coleman2017dawnbench}.

\section{The Design and Implementation of Application Benchmark}\label{methodology}

On the basis of the AIBench framework illustrated in Section~\ref{framework}, we implement the first end-to-end AI application benchmark, modelling the complete use-cases of a realistic E-commerce search intelligence.

\subsection{The Design and Implementation}

\begin{figure}[tb]
\centering
\includegraphics*[scale=0.68]{aibench-arch.pdf}
\caption{AIBench Implementation.}
\label{ecommerce-arch}
\end{figure}

The end-to-end application benchmark consists of four modules: online server, offline analyzer, query generator, and data storage, as shown in Fig.~\ref{ecommerce-arch}. Among them, online server receives the query requests and performs personalized searching and recommendation, integrated with AI inference.

Offline analyzer chooses the  appropriate AI  algorithm implementations and performs a training stage to generate a learning model. Also, the offline analyzer is responsible to build data indexes to accelerate data access.

Query generator is to simulate concurrent users and send query requests to online server based on a specific configuration. The configuration designates parameters like concurrency, query arriving rate,  distribution, and user thinking time, to simulate different query characteristics and satisfy multiple generation strategies. We implement our query generator based on JMeter~\cite{jmeter2017apache}.

Data storage module stores all kinds of data, including the user database that saves all the attributes of user information, the product database that holds all the attributes of product information, logs that record the complete query histories, text data that contains the product description text or the user comments, image and video data that depict the appearance and usage of product vividly, and audio data that stores the voice search data and voice chat data. Overall, the data storage covers various data types including structured, un-structured, and semi-structured data, and diverse data sources, including table, text, image, audio and video.

To support scalable deployment on the clusters with different scales, each module is able to deploy on multiple nodes. Also, a series of data generators are provided to generate the e-commerce data with different scales, through setting several parameters, e.g., the number of products and product attribute fields, the number of users and user attribute fields.

\subsubsection{Online Server}

Online server provides personalized searching and recommendations combining traditional machine learning and deep learning technologies. Online server consists of four submodules including search planer, recommender, searcher, and ranker.

\textbf{Search Planer} is the entrance of online server.  It is responsible for receiving the query requests from query generator, and sending the request to the other online components and receiving the return results. We use the Spring Boot framework~\cite{webb2013spring} to implement the search planer.

\textbf{Recommender} is to analyze the query item and provides personalized recommendation, according to the user information obtained from the user database. It first conducts query spelling correction and query rewrite, then it predicts the belonged category of the query item based on a classification model---FastText~\cite{joulin2016fasttext}. Using a deep neural network proposed by Alibaba ~\cite{ni2018perceive}, the query planer then conducts an inference process and uses the offline trained model to provide personalized recommendation. It returns two vectors---the probability vector of the predicted categories and the user preference score vector of product attribute, such as the user preference for brand, color, etc.
We use the Flask Web framework~\cite{grinberg2018flask} and Nginx~\cite{reese2008nginx} to build our recommender for category prediction, and adopt TensorFlow serving~\cite{olston2017tensorflow} to implement online recommendation.

To guarantee scalability and service efficiency, Searcher follows an industry scale architecture. \textbf{Searcher} is deployed on several different, i.e., three  clusters, that hold the inverted indexes of product information in memory to guarantee high concurrency and low latency. In view of the click-through rate and purchase rate, the products belong to three categories according to the popularity---high, medium, and low, occupying the proportion of 15\%, 50\%, and 50\%, respectively. Note that the high popularity category is the top 15\% popular products chosen from the medium popularity category.  The indexes of products with different popularities are stored into different clusters.
Given a searching request, the searcher searches these three clusters one by one, until reaching a specific amount. Generally, the cluster that holds low popularity products is rarely searched in a realistic scenario.
So for each category, searcher adopts different deployment strategies. The cluster for high popularity contains more nodes and more backups to guarantee the searching efficiency. While the cluster for low popularity deploys the least number of nodes and backups. We use the Elasticsearch~\cite{gormley2015elasticsearch} to set up and manage the three clusters of searcher.

\textbf{Ranker} uses the weight returned by \emph{Recommender} as initial weight, and ranks the scores of products through a personalized L2R neural network~\cite{ni2018perceive}. The ranker also uses the  Elasticsearch~\cite{gormley2015elasticsearch} to implement product ranking.

The online serving process is as follows.

(1) \emph{Query Generator} simulates concurrent users and sends query requests to \emph{Search Planer};

(2) \emph{Search Planer} receives the query request and sends the query item to \emph{Recommender};

(3) \emph{Recommender} analyzes the query and returns the category prediction results and personalized attribute weights to \emph{Search Planer};

(4) \emph{Search Planer} sends the initial query item and predicted category results to \emph{Searcher};

(5) \emph{Searcher} searches the inverted indexes and returns the product ID to \emph{Search Planer};

(6) \emph{Search Planer} sends the product ID and personalized attribute weights to \emph{Ranker};

(7) \emph{Ranker} ranks the product according to the initial weights and returns the ranking scores to \emph{Search Planer};

(8) \emph{Search Planer} sends a product database access request according to the product ID and obtains the product information;

(9) \emph{Search Planer} returns the searched product information to \emph{Query Generator}.

\subsubsection{Offline Analyzer}

Offline analyzer is responsible for training models and building indexes to improve the online serving performance. It consists of three parts---AI trainer, job scheduler, and indexer.

AI trainer is to train models using related data stored in database. To dig the features within product data, e.g., text, image, audio, video, and power the efficiency of online server, the AI trainer chooses ten AI algorithms (component benchmarks) from the AIBench framework currently, including classification  for category prediction, recommendation  for personalized recommendation, learning to ranking  for result scoring and ranking, image-to-text  for image caption, image-to-image and image generation for image resolution enhancement, face embedding  for face detection within an image, spatial transformer  for image rotating and resizing, object detection  for detecting video data, speech recognition for audio data recognition.

Job schedule provides two kinds of training mechanisms: batch processing and streaming-like processing. In a realistic scenario, some models need to be updated frequently. For example, when we search an item and click one product showed in the first page, the application will immediately train a new model based on the product that we just clicked, and make new recommendations shown in the second page. Our benchmark implementation considers this situation, and adopts a streaming-like approach to update the model every several seconds. For batch processing, the trainer will update the models every several hours.
The indexer is to build indexes for product information. In total, the indexer provides three kinds of indexes, including the inverted indexes with a few fields of products for searching, forward indexes with a few fields for ranking, and forward indexes with a majority of fields for summary generation.

\subsection{Extensibility for Other Industry Applications}

Taking medicine AI scenario as an example---which is one of the most representative scenario for AI, we illustrate how we use the AIBench framework to construct an end-to-end benchmark for clinical diagnosis application.
The AI-related critical path of clinical diagnosis contains the following steps. 1) train a series of diagnosis models according to history data offline, e.g., detection model, classification model, recommendation model, etc; 2) detect the abnormal information within patient's physical examination data, such as the tumour detection of a CT image; 3) classify and predict the potential disease which is an online inference stage; 4) recommend an optimal treatment plan.

To construct an end-to-end clinical diagnosis application benchmark, the AIBench framework is flexible to provide an AI-related offline module and online module. Within the offline module, component benchmarks of object detection, classification, and recommendation are chosen for training models. Within the online module, these models are loaded for online inference as a service.

In conclusion, the AIBench framework is extensible for building other industry applications.

\section{Experiment Setup}

In this section, we describe the experimental setting and methodology.

\subsection{Node Configurations}

We deploy a 16-node CPU and GPU cluster.
For the CPU cluster, all the nodes are connected with a 1 Gb ethernet network. Each node is equipped with two Xeon E5645 processors and 32 GB memory. Each processor contains six physical out-of-order cores. The OS version of each node is Linux CentOS 6.9 with the Linux kernel version 3.11.10. The software versions are JDK 1.8.0, Python 3.6.8, and GCC 5.4, respectively.
The GPU node is equipped with four Nvidia Titan XP GPUs. Every Titan XP owns 3840 Nvidia Cuda cores and 12 GB memory.
The detailed hardware configuration of each node is listed in Table~\ref{hwconfigeration}.

\begin{table}
\caption{Hardware Configuration Details.}\label{hwconfigeration}
\renewcommand\arraystretch{1.1}
\center
\scriptsize
\begin{tabular}{|p{1in}|p{1in}|p{1in}|p{1in}|}
\hline \rowcolor{mygray} \multicolumn{4}{|l|}{CPU Configurations}\\
\hline \multicolumn{2}{|c|}{CPU Type} & \multicolumn{2}{c|}{Intel CPU Core} \\
\hline \multicolumn{2}{|c|}{Intel \textregistered Xeon E5645}  &\multicolumn{2}{c|}{6 cores@2.40G} \\
\hline L1 DCache &L1 ICache &L2 Cache &L3 Cache \\
\hline 6 $\times$ 32 KB& 6 $\times$ 32 KB&6 $\times$ 256 KB& 12MB \\
\hline \multicolumn{2}{|c|}{Memory} & \multicolumn{2}{c|}{32GB, DDR3}\\
\hline \multicolumn{2}{|c|}{Ethernet} & \multicolumn{2}{c|}{1Gb}\\
\hline \multicolumn{2}{|c|}{Hyper-Threading} & \multicolumn{2}{c|}{Disabled}\\
\hline \rowcolor{mygray} \multicolumn{4}{|l|}{GPU Configurations}\\
\hline \multicolumn{2}{|c|}{GPU Type} & \multicolumn{2}{c|}{Nvidia Titan XP} \\
\hline \multicolumn{2}{|c|}{Nvidia Cuda Cores} & \multicolumn{2}{c|}{3840 cores} \\
\hline \multicolumn{2}{|c|}{GPU Memory} & \multicolumn{2}{c|}{12GB, GDDR5X} \\
\hline

\end{tabular}
\end{table}

\subsection{The Benchmark Deployment}

We deploy the end-to-end AI application benchmark, introduced in Section~\ref{methodology} on the above mentioned clusters, for online server and offline trainer.

\textbf{Online Server Setting.}
Online server is deployed on the 16-node CPU cluster, containing one query generator node, one search planer node, two recommender nodes, nine searcher nodes, one ranker node, and two nodes for data storage. The detailed module setup information and involved software information are listed in Table~\ref{server-setup}.

\begin{table}
\caption{Online Server Settings.}\label{server-setup}
\renewcommand\arraystretch{1.1}
\center
\scriptsize
\begin{tabular}{|p{1in}|p{1.5in}|p{0.8in}|p{1.5in}|}
\hline \multicolumn{2}{|c|}{Module} &  Nodes & Software \\
\hline \multicolumn{2}{|c|}{Query Generator} & 1 & Jmeter 5.1.1  \\
\hline \multicolumn{2}{|c|}{Search Planer} & 1 & SpringBoot 2.1.3  \\
\hline \multirow{2}{*}{Recommender} & Category Predictor & 1 & Flask Web 1.1.1, Nginx 1.10.3   \\
\cline{2-4} & TensorFlow Serving & 1 & TensorFlow Serving 1.14.0 \\
\hline \multirow{3}{*}{Searcher} & Cluster (high popularity) & 4 & \multirow{3}{*}{Elasticsearch 6.5.2} \\
\cline{2-3} & Cluster (medium popularity) & 3 &  \\
\cline{2-3} & Cluster (low popularity) & 2 &  \\
\hline \multicolumn{2}{|c|}{Ranker} & 1 & Elasticsearch 6.5.2 \\
\hline \multirow{2}{*}{Data Storage} & User database & 1 & Neo4j 3.5.8  \\
\cline{2-4}  & Product database & 1 & Elasticsearch 6.5.2  \\
\hline

\end{tabular}
\end{table}

\textbf{Offline Trainer Settings.}
Offline trainer is deployed on the GPUs. The CUDA and Nvidia driver versions are 10.0 and 410.78, respectively. We evaluate the PyTorch implementations with the version of 1.1.0. The data sets for each benchmark are ImageNet for image classification (137 GB), LSUN for image generation (42.8 GB), VGGFace2 for face embedding (36 GB), Microsoft COCO (13 GB) for Image-to-Text and object detection, MNIST (9.5 MB) for spatial transformer, Cityscapes (267 MB) for Image-to-Image, MovieLens (190 MB)  for recommendation, Librispeech (59.3 GB) for speech recognition, and Gowalla (107 MB) for learning to rank, respectively.

\subsection{Performance Data Collection}

We use network time protocol (NTP)~\cite{mills1985network} to perform clock synchronization in all cluster nodes and obtain the latency and tail latency metrics of online server.
We use a profiling tool---Perf~\cite{de2010new} to collect the CPU micro-architectural data through hardware performance monitoring counters (PMCs).
For GPU profiling, we use Nvidia profiling toolkit---nvprof~\cite{nvprof} to track the running performance of GPU.
To obtain more accurate metric numbers, we run each benchmark three times and report the average value.

\section{Evaluation}
In this section, we evaluate the end-to-end AI application benchmarks introduced in Section~\ref{methodology}, including online server and ten AI component benchmarks included in offline analyzer.

\subsection{Evaluation of Online Server}

We evaluate the online server performance on the 16-node CPU cluster. The product database contains a hundred thousand products with 32 attribute fields.
The query generator simulates 1000 users with 30-second warm up time. The users send query requests continuously every think time interval, which follows a poisson distribution. In total, we collect the performance numbers when 20,000 query requests finish.

\begin{figure}[tb]
\centering
\includegraphics*[scale=0.53]{latency.pdf}
\caption{Latency of Online Server.}
\label{latency}
\end{figure}

Latency is an important metric to evaluate the service quality.
Fig.~\ref{latency} shows the latencies of online server.  We find that the overall latencies of the entire execution paths of the current baseline implementation are 161.13, 392, and 956 milliseconds for the average, 90th percentile and 99th percentile latencies respectively, as listed in Fig.~\ref{latency}(a)~\footnote{With respect to the real numbers in our industry partner, the number is quite high. They have taken many measures to decrease the overall latency. However, this baseline implementation indeed confirm the importance of the AI components in the critical paths as the data access and communication overhead can be further decreased.}.
We further deeply analyze the latency of each module, including recommender, searcher, and ranker, as shown in Fig.~\ref{latency}(b). The latency of search planer is negligible, so we do not report it in Fig.~\ref{latency}(b). We find that the recommender occupies the largest latency: 75.7 milliseconds, 209.4 milliseconds, and 557.7 milliseconds for the average, 90th percentile, 99th percentile latencies, respectively. In comparison, the latencies of searcher and ranker are both within 4 milliseconds. Although recommender and ranker both contain AI-related components, they incur different latencies. The average communication latency between the modules is also high, nearly equal to that of recommender.

Furthermore,  Fig.~\ref{latency}(c) presents the latency breakdown of recommender in terms of query parsing, user database access, category prediction, and TensorFlow serving.
We find that database access  and TensorFlow serving latencies  are the top two factors that impact the service performance. The user information is represented as graph data that contains entities and relationships. The sophisticated data structure and frequent garbage collection may influence the data access speed largely. While TensorFlow serving needs to run a forward inference using the recommendation model, and thus incurs larger latency. In order to measure the impact of the AI component on service performance and find the bottlenecks, we make a discussion from the following aspects.

\textbf{The weights of AI-related components on service performance.} AI components change the critical path significantly. In our evaluation, the time spent on AI-related and non AI-related components is 34.29 and 49.07 milliseconds for the average latency, 74.8 and 135.7 milliseconds for the 90th percentile latency, 152.2 and 466.5  milliseconds for the 99th percentile latency, except for the data preprocessing and communication latency, which indicates that an industry scale AI application benchmark suite is essential to depict the characteristics of a modern Internet service.

\textbf{The limitations for AI to ensure a good service.} The online inference module needs to load the trained model and conducts a forward computation to obtain the result. However, the depth or the size of a neural network model may largely affect the inference time. For comparison, we train a more complicated neural network for TensorFlow serving with the size of the model increasing from 184 MB to 253 MB. We find that the latency of TensorFlow serving increases sharply, with the average latency increasing from 30.78 to 125.71 milliseconds, and the 99th percentile latency increasing from 149.12 to 5335.12 milliseconds. Hence, the Internet service architects must perform a trade-off between the service quality and the depth or size of a neural network model.

\textbf{The difference of micro-architectural behaviors.} We characterize the changes of micro-architectural behaviors from the following two aspects.

\begin{itemize}
\item \textbf{The difference between non AI-related and AI-related components.} We use perf to sample the cache behaviors of AI-related and non AI-related components. We find that comparing to the non AI-related components, the AI-related components suffer from more cache misses in memory hierarchy, especially L2 cache misses per Kilo instructions. TensorFlow serving suffers from 61 L2 cache misses per Kilo instructions, while the average number of non AI-related components is 37.

\item \textbf{The changes from a small neural network model to a large one.} We compare two neural network models for TensorFlow serving, with a smaller one 184 MB, and a larger one 253 MB. We also sample their cache behaviors. We find that with a larger model, the L3 cache misses per Kilo instructions increase extremely sharply, from 1.38 to 8.9, which incurs large memory backend stalls to fetch data from memory. And thus the 99th percentile latency increases more than thirty times.

\end{itemize}

\subsection{Evaluation of Offline Training}

\begin{figure}[tb]
\centering
\includegraphics*[scale=1.1]{sm.pdf}
\caption{SM Efficiency.}
\label{sm}
\end{figure}

GPU architecture contains multiple streaming multiprocessors (SM), each of which has a certain number of CUDA cores, memory registers, memory caches,  warp schedulers, etc.
The GPU efficiencies of running AI benchmarks are significantly important for both the GPU architecture design and AI system optimization. In this subsection, we mainly explore the GPU execution efficiency and evaluate the ten component benchmarks used in offline analyzer of the end-to-end AI application benchmark on Titan XP GPU. We choose the PyTorch implementations with the version of 1.1.0 for evaluation.

We comprehensively characterize the GPU efficiency, drilling down to functional-level running time breakdown and execution stall analysis. The overall execution performance of these ten component benchmarks are varying in terms of SM efficiency, which measures the percentage of time that the SM has one or more warps are active. Fig.~\ref{sm} shows the SM efficiency of each benchmark, with the value ranging from 29\% (Learning\_to\_rank) to 95\% (Recommendation). We find that some benchmarks like learning\_to\_rank have extremely low SM efficiency comparing to the other benchmarks. To discover the factors that impact the performance greatly, we first conduct running time breakdown analysis and decompose the benchmarks to the hotspot kernels or functions, then we find the GPU execution efficiency in terms of different percentage of stalls.

\subsubsection{Running Time Breakdown}\label{timebreakdown}

\begin{figure}[tb]
\centering
\includegraphics*[scale=0.9]{time-kernel.pdf}
\caption{Running Time Breakdown of the Ten Component Benchmarks.}
\label{time-breakdown}
\end{figure}

We use nvprof to trace the running time breakdown and find the hotspot functions that occupy more than 80\% of running time in total.
Since each run involves  dozens of function calls, we single out the functions that occupy large proportions of running time and classify them into several categories of kernels according to their computation logics.
Through statistics, we find that the most time-consuming functions among all the ten component benchmarks have much in common, and they are classified into six categories of kernels: convolution, general matrix multiply (gemm), batch normalization,  relu activation, element-wise operation and gradient calculation, which is consistent with our micro benchmarks and further indicates the decision of including them is correct. Note that each kernel contains a bunch of functions that solve the similar problem. For example, gemm kernel includes single or double precision floating general matrix multiply, etc.
Fig.~\ref{time-breakdown} shows the running time breakdown of the above six kernels, using the average value of all involved functions within each kernel. Note that the remaining 20\% functions are not considered in this figure.
Further, for each kernel, we summarize the typical functions that occupy a large proportion of running time among the ten component benchmarks, as shown in Table~\ref{func-sum}. We find that learning\_to\_rank spends too much time on the convolution category, with the mainstream function call of maxwell\_scudnn\_128x32\_stridedB\_splitK\_interior\_nn, whose SM efficiency is only 18.5\%, so this is the reason why leaning\_to\_rank has the lowest SM efficiency of 29\%.
We believe that the six kernels and these corresponding functions are the optimization directions not only for CUDA library optimizations but also for micro-architectural optimizations.

\begin{table}[htbp]
\caption{Hotspot Functions for Each Kernel.}
\renewcommand\arraystretch{1.2}
\scriptsize
\label{func-sum}
\center
\begin{tabular}{|p{1in}|p{4in}|}
\hline
\textbf{Kernel} & \textbf{Function Name} \\
\hline
\multirow{3}{*}{Convolution} &  maxwell\_scudnn\_128x128\_stridedB\_splitK\_interior\_nn \\
\cline{2-2}
& maxwell\_scudnn\_128x32\_stridedB\_splitK\_interior\_nn\\
\cline{2-2}
 & maxwell\_scudnn\_winograd\_128x128\_ldg1\_ldg4\_tile148n\_nt \\
\hline
\multirow{3}{*}{GEMM} & maxwell\_sgemm\_128x64\_nt \\
\cline{2-2}
& maxwell\_sgemm\_128x64\_nn \\
\cline{2-2}
& sgemm\_32x32x32\_NN\_vec \\
\hline
\multirow{3}{*}{BatchNorm} &  cudnn::detail::bn\_fw\_tr\_1C11\_kernel\_NCHW  \\
\cline{2-2}
& cudnn::detail::bn\_bw\_1C11\_kernel\_new  \\
\cline{2-2}
& batch\_norm\_backward\_kernel\\
\cline{2-2}
& at::native::batch\_norm\_backward\_kernel \\
\hline
\multirow{2}{*}{Relu} & maxwell\_scudnn\_128x128\_relu\_small\_nn  \\
\cline{2-2}
& maxwell\_scudnn\_128x128\_relu\_small\_nn  \\
\cline{2-2}
& maxwell\_scudnn\_128x32\_relu\_interior\_nn \\
\hline
\multirow{2}{*}{\tabincell{l}{Element-wise}} & element-wise add kernel \\
\cline{2-2}
& element-wise threshold kernel \\
\hline
\multirow{2}{*}{Gradient} & cudnn::detail::dgrad\_engine \\
\cline{2-2}
& cudnn::detail::dgrad\_alg1\_engine \\
\hline

\end{tabular}
\end{table}

\subsubsection{GPU Execution Efficiency Analysis}

Focusing on the above six most time-consuming kernels, we evaluate the stalls of these kernels, including instruction fetch stall (Inst\_fetch), which indicates the percentage of stalls because the next assembly instruction has not yet been fetched, execution dependency stall (Exe\_depend), which is the percentage of stalls because an input required by the instruction is not yet available, memory dependency stall (Mem\_depend), which is the percentage of stalls because a memory operation cannot be performed due to the required resources not being available or fully utilized, texture stall (Texture), which is the percentage of stalls because of the under-utilization of the texture sub-system,  synchronization stall (Sync), which is the percentage of stalls due to a syncthreads call, constant memory dependency stall (Const\_mem\_depend), which is the percentage of stalls because of immediate constant cache miss, pipe busy stall (Pipi\_busy), which is percentage of stalls because a compute operation cannot be performed because the compute pipeline is busy, and memory throttle stall (Mem\_throttle), which is the percentage of stalls due to large pending memory operations~\cite{nvprof}.

\begin{figure}[tb]
\centering
\includegraphics[scale=1]{stall.pdf}
\caption{Stall Breakdown of Each Kernel.}
\label{stall-breakdown}
\end{figure}

The breakdown of eight kinds of stalls of each kernel is shown in Fig.~\ref{stall-breakdown}.
We find that the top two GPU execution stalls are memory dependency stalls, and execution dependency stalls. For example, for Element-Wise kernels, the memory dependency stalls occupy a very large proportion of 68\%, thus resulting low SM efficiency of about 50\%.
The memory dependency stalls may occurs due to the high cache misses and thus the load/store resources are not available. The optimization strategies include optimizing date alignment, data locality, and data access pattern. The execution dependency stalls may occur due to the low instruction-level parallelism, so exploiting ILP may alleviate partial execution dependency stalls to a certain degree.

We also identify the functional level stalls, including the hotspot functions
 illustrated in Table~\ref{func-sum}, to provide potential optimization guidelines for function calls. We find that performing analytics of stalls on a function level in addition to a kernel level is helpful. For example, the memory dependency stall percentage of maxwell\_scudnn\_128x32\_stridedB\_splitK\_interior\_nn function in ``convolution" category achieves 61\%, however, the percentage of maxwell\_sgemm\_128x64\_nn function in ``GEMM" category is 18\%, indicating that different optimization strategies are needed to achieve maximum performance improvement.

\section{Conclusion}

This paper presents the first industry standard Internet service AI benchmark suite with seventeen industry partners.
We propose and implement a highly extensible, configurable, and flexible AI benchmark framework, and identify sixteen prominent AI problem domains from three most important Internet services domains: search engine, social network, and e-commerce.
On the basis of the AIBench framework, we
design and implement the first end-to-end Internet service AI benchmark suite, with an
underlying e-commerce searching model.
 On the CPU and GPU clusters, we perform a preliminary evaluation of the end-to-end application benchmark. We observe that the AI-related components significantly change the
critical paths and workload characterization of  the Internet service, which justifies the end-to-end AI application benchmark.

redpajama_set_name: RedPajamaArXiv


This reversible pebble leather belt gives you two looks in one: black on one side and brown on the other. The fit is completely customizable—just cut the leather with scissors. It comes packaged and ready to gift in a signature Coach box.


Published 04/25/2019 04:44:22 pm at 04/25/2019 04:44:22 pm in Parsons Style Desk.
parsons style desk west elm walnut parsons style desk .
white corner desk with drawers mekongdetalclub white corner desk with drawers white or black simple and minimal parsons style desk with drawer, walmart altra furniture parsons style desk with drawer and walmart altra furniture parsons style desk with drawer and bookcase black oak, altra parsons desk black parsons desk parsons desk urban salted pine altra parsons desk parsons desk with drawers parsons desk by sandalwood parsons desk with drawer white altra parsons desk , altra furniture parsons style desk with drawer white and teal altra furniture parsons style desk with drawer white and teal http, , black glass marble parsonsstyle desk chairish black glass marble parsonsstyle desk, parsons desk with drawer white staples altra furniture parsons style desk with drawer and bookcase black oak finish, desk with drawer pallet desk with drawers and shelves pallet desk , how to style the west elm parsons desk the everygirl the west elm parsons desk search for it on pinterest we dare you it is with great confidence that we state this is the most popular desk on the market , parsons desks parsons desks with drawers hansflorineco parsons desks parsons style desk with drawers parsons desks , parsons style burlwood henredon desk chairish contemporary parsons style burlwood henredon desk for sale image of .


Cape Town - With New Zealand having safely banked eight log points in the Castle Rugby Championship against arch-rivals Australia, the Springboks have to urgently ask themselves ahead of their return assignment against Argentina in Mendoza on Saturday night (21:10): will eight of our own be enough against Argentina?
It is a complex issue, because while very few teams can ever tackle the Pumas in their own den genuinely confident that they will even secure the basic victory, never mind any broader aspirations, the importance of nicking at least one bonus point against the tourney newcomers has also come into fresh focus.
Let's face it, the World Cup champion All Blacks - already pretty clear favourites for the revamped southern hemisphere competition before it began - would be sitting especially snugly after completion of a third of the fixtures if the Boks can do no more in Argentina than draw level on points with Richie McCaw's side on the ladder via a "no cherry on top" triumph.
So, having failed by an agonising one touchdown to get the four-try bonus point at Newlands last weekend, South Africa must somehow attempt to make up for it by striking out for a full house in Mendoza.
At least then, after completing arguably their two least daunting fixtures on paper, the Boks would have the minor psychological comfort of having their noses in front of an All Blacks side who have just retained the Bledisloe Cup for the umpteenth time and simultaneously finished their Championship obligations against a Wallabies outfit still ranked second in the world by a whisker.
The fixture list only gets tougher for the Boks, by contrast, after the Mendoza date, with four remaining fixtures against their established pair of SANZAR partners.
There is another incentive for Jean de Villiers and company to hit their straps in Argentina: according to the official IRB website, South Africa will move ahead of Australia into second by a fraction on Saturday night if they can beat the Pumas by more than 15 points.
Meanwhile, many observers and analysts both in New Zealand and elsewhere would only have been left suspecting the All Blacks will be desperately hard to outrun to the 2012 title, courtesy of their consummate 22-0 win against the Wallabies at Eden Park.
Don't be too deceived by - or seek too much solace in - the fact that the home side only managed one try.
It was an illuminatingly convincing performance in most other respects, with the All Blacks moving up a discernible gear after their slightly more laboured, low-key victory a week earlier in Sydney.
In short, they did not give the Aussies (for whom Kiwi-born coach Robbie Deans must be coming under mounting pressure) a sniff in the game, as reflected by the unflattering "nil" to their scoring column.
You had to give the Wallabies some credit for their stubborn defensive resilience, under the circumstances, but otherwise it was sobering, one-way traffic.
"The whole game just seemed like sweeping movements of All Blacks," Tri-Nations-winning former Springbok coach Nick Mallett said admiringly in the SuperSport studio.
Some consolation will be the knowledge that one of their essential backline string-pullers, Sonny Bill Williams, now bids goodbye to rugby union for the foreseeable future - he was no less influential in this swansong, though not necessarily permanently so, appearance.
Co-pundit Naas Botha also made the point that New Zealand's scrummaging consistency and power seems to have dipped a little, and that their lineout - with Keven Mealamu's throwing-in wonky until his substitution - is another area the Boks may be able to target when they first run into the world champions this year in Dunedin on September 15.


text:
Björn Gunnlaugsson (* 25. September 1788 auf dem Hof Tannstaðir; † 17. März 1876 in Reykjavík) war ein isländischer Kartograf, der die erste vollständige Karte seiner Heimatinsel anfertigte.

Björn Gunnlaugsson bereiste im Auftrag der Literarischen Gesellschaft Islands (Hið íslenska bókmenntafélag) von 1831 bis 1843 Island, die Ergebnisse seiner Arbeit wurden als topografische Karte auf vier Blättern mit einem Maßstab von 1:480.000 veröffentlicht. Als Zeitpunkt wird oft 1844 angegeben, wahrscheinlich wurden die Karten aber nicht vor 1848 fertiggestellt. 1849 folgte eine kleinere Ausgabe der Karte mit einem Maßstab von 1:960.000.

Literatur

 C. F. Bricka (Hrsg.): Dansk biografisk Lexikon: tillige omfattende Norge for Tidsrummet 1537–1814. Gyldendal, Kjøbenhavn 1887–1905. Bd. 12.

Weblinks

 National and University Library of Island: The Survey of Iceland – The maps of Björn Gunnlaugsson.

Kartograf (19. Jahrhundert)
Isländer
Geboren 1788
Gestorben 1876
Mann

redpajama_set_name: RedPajamaWikipedia


text:
App Store Market Worth Nearly $2.5 Billion Per Year?
Thursday August 27, 2009 8:43 am PDT by Eric Slivka
Mobile advertising firm AdMob has released its July 2009 Mobile Metrics report along with a supplemental presentation in which the company reveals the results of a survey of over 1,100 iPhone, iPod touch, and Android users to obtain data on application usage. The study reveals that iPhone users download approximately 10 applications per month on average with 25% of those being paid applications. iPod touch users download over 18 applications per month on average, with only two of those being paid applications.
The study also found that approximately 50% of iPhone users download at least one paid application per month, while only 40% of iPod touch users do likewise. Users who do download paid applications average approximately five paid downloads per month and spend an average of $9.49 (iPhone) or $9.79 (iPod touch) per month for an average application price of just under $2.00.
Extrapolating the survey data to the estimated user bases of 26.4 million iPhone users and 18.6 million iPod touch users, AdMob suggests that the total App Store market size may be nearly $200 million per month, or approaching $2.5 billion per year. In contrast, the Android market is estimated at approximately $60 million per year.
AdMob's survey method of soliciting respondents via ads within its own network, which serves 7,000 mobile web sites and 3,000 applications, may skew the survey results somewhat by attracting users more likely to be engaged in mobile applications, but the results do still provide an idea of the substantial market for App Store developers and Apple itself.

redpajama_set_name: RedPajamaCommonCrawl


1. The above results, after approval by audit committee and audited by Statutory Auditors of the company have been taken on record by Board of Directors at its meeting held on 13th May, 2011.
2. The Company is exclusively engaged in the Pharmaceutical Business segment.
3. There were no investor complaints pending at the beginning of the quarter. During the quarter, 07 shareholder's complaints were received and resolved.
4. Statement of assets and liabilities as on 31.03.2011.


text:
Robust Linear Transceiver Designs for Vector Parameter Estimation in MIMO Wireless Sensor Networks under CSI Uncertainty
Rajput, K. P., Verma, Y., Venkategowda, N. K. D., Jagannatham, A. K. & Varshney, P. K., Aug 2021, In: IEEE Transactions on Vehicular Technology. 70, 8, p. 7347-7362 16 p., 9444772.
Channel State Information 100%
Wireless Sensor Networks 76%
Channel state information 53%
Transceivers 48%
Robust MIMO Communications Against Antenna Blockage and Interference
Liu, Y., Chen, B., Mroczek, J. J., Malowicki, J. E. & Michalak, R. J., Nov 2019, MILCOM 2019 - 2019 IEEE Military Communications Conference. Institute of Electrical and Electronics Engineers Inc., 9020916. (Proceedings - IEEE Military Communications Conference MILCOM; vol. 2019-November).
Software radio 64%
Robustness of change detection algorithms in the presence of registration errors
Sundaresan, A., Varshney, P. K. & Arora, M. K., Apr 2007, In: Photogrammetric Engineering and Remote Sensing. 73, 4, p. 375-383 9 p.
Robustness of feedforward neural networks
Chiu, C. T., Mehrotra, K. G., Mohan, C. K. & Ranka, S., 1993, IEEE International Conference on Neural Networks - Conference Proceedings. Institute of Electrical and Electronics Engineers Inc., Vol. 1993-January. p. 783-788 6 p. 298655
Feedforward neural networks 100%
Neural networks 54%
Disasters 42%
Robustness of the Counting Rule for Distributed Detection in Wireless Sensor Networks
Goel, A., Patel, A., Nagananda, K. G. & Varshney, P. K., Aug 2018, In: IEEE Signal Processing Letters. 25, 8, p. 1191-1195 5 p.
Target 57%
Robustness 56%
Robust occupancy-based distributed demand control ventilation
Dhummi, V., Demetriou, D., Palanthandalam-Madapusi, H. J., Khalifa, H. E. & Isik, C., Mar 2011, In: International Journal of Ventilation. 9, 4, p. 359-369 11 p.
Ventilation 100%
Energy utilization 31%
Robust Power Allocation for Resource-Aware Multi-Target Tracking with Colocated MIMO Radars
Yuan, Y., Yi, W., Hoseinnezhad, R. & Varshney, P. K., 2021, In: IEEE Transactions on Signal Processing. 69, p. 443-458 16 p., 9309315.
Radar 9%
Quality of service 9%
Robust resource provisioning in time-varying edge networks
Yu, R., Xue, G., Wan, Y., Tang, J., Yang, D. & Ji, Y., Oct 11 2020, MobiHoc 2020 - Proceedings of the 2020 International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing. Association for Computing Machinery, p. 21-30 10 p. (Proceedings of the International Symposium on Mobile Ad Hoc Networking and Computing (MobiHoc)).
Stochastic models 44%
NP-hard 39%
Robust sparse Bayesian learning STAP method for discrete interference suppression in nonhomogeneous clutter
Sun, Y., Yang, X., Long, T. & Sarkar, T. K., Jun 7 2017, 2017 IEEE Radar Conference, RadarConf 2017. Institute of Electrical and Electronics Engineers Inc., p. 1003-1008 6 p. 7944350
space-time adaptive processing 100%
Space time adaptive processing 68%
clutter 66%
Robust suboptimal decision fusion in wireless sensor networks
Jiang, R., Misra, S., Chen, B. & Swami, A., Dec 1 2005, MILCOM 2005: Military Communications Conference 2005. 1605981. (Proceedings - IEEE Military Communications Conference MILCOM; vol. 2005).
Fusion reactions 100%
Role of Academia to Create Re-entry Pathways in Computing
Billionniere, E., Rahman, F., Brown, Q. & Seo, H., Mar 10 2020, 2020 Research on Equity and Sustained Participation in Engineering, Computing, and Technology, RESPECT 2020 - Proceedings. Gardner-McCune, C., Washington, N., Dillon, E., Washington, G. & Payton, J. (eds.). Institute of Electrical and Electronics Engineers Inc., 9272425. (2020 Research on Equity and Sustained Participation in Engineering, Computing, and Technology, RESPECT 2020 - Proceedings).
occupational reintegration 100%
Education computing 31%
student 18%
Room shape reconstruction with a single mobile acoustic sensor
Peng, F., Wang, T. & Chen, B., Feb 23 2016, 2015 IEEE Global Conference on Signal and Information Processing, GlobalSIP 2015. Institute of Electrical and Electronics Engineers Inc., p. 1116-1120 5 p. 7418371
Loudspeakers 51%
Time of arrival 51%
Routing wires with non-uniform width and spacing in data paths
Krishna, B., Chen, C. Y. R. & Sehgal, N. K., 1999, Proceedings of the International Conference on Microelectronics, ICM. Institute of Electrical and Electronics Engineers Inc., Vol. 2000-January. p. 85-88 4 p. 884811
Routing algorithms 27%
Heuristic algorithms 26%
RSS-Based Detection of Drones in the Presence of RF Interferers
Sinha, P., Yapici, Y., Guvene, I., Turgut, E. & Cenk Gursoy, M., Jan 2020, 2020 IEEE 17th Annual Consumer Communications and Networking Conference, CCNC 2020. Institute of Electrical and Electronics Engineers Inc., 9045281. (2020 IEEE 17th Annual Consumer Communications and Networking Conference, CCNC 2020).
radio 100%
Rule networks in learning classifier systems
Kuber, K., Card, S. W., Mehrotra, K. G. & Mohan, C. K., 2014, GECCO 2014 - Companion Publication of the 2014 Genetic and Evolutionary Computation Conference. Association for Computing Machinery, p. 977-982 6 p. (GECCO 2014 - Companion Publication of the 2014 Genetic and Evolutionary Computation Conference).
Learning Classifier Systems 100%
Classifiers 34%
Set Convergence 18%
Experimentation 14%
Running ASCEND, DESCEND and PIPELINE algorithms in parallel using small processors
Vaidyanathan, R., Hartmann, C. R. P. & Varshney, P. K., Apr 29 1993, In: Information Processing Letters. 46, 1, p. 31-36 6 p.
SAFE: Similarity-Aware Multi-modal Fake News Detection
Zhou, X., Wu, J. & Zafarani, R., 2020, Advances in Knowledge Discovery and Data Mining - 24th Pacific-Asia Conference, PAKDD 2020, Proceedings. Lauw, H. W., Lim, E-P., Wong, R. C-W., Ntoulas, A., Ng, S-K. & Pan, S. J. (eds.). Springer, p. 354-367 14 p. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); vol. 12085 LNAI).
Vision 100%
Similarity 83%
Relationships 44%
Sampling community structure in dynamic social networks
Mensah, H. & Soundarajan, S., 2018, Recent Trends and Future Technology in Applied Intelligence - 31st International Conference on Industrial Engineering and Other Applications of Applied Intelligent Systems, IEA/AIE 2018, Proceedings. Ait Mohamed, O., Mouhoub, M., Sadaoui, S. & Ali, M. (eds.). Springer Verlag, p. 114-126 13 p. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); vol. 10868 LNAI).
Community Structure 100%
Dynamic Networks 94%
Social Networks 79%
Sampling 46%
Query 32%
Sampling dark networks to locate people of interest
Wijegunawardana, P., Ojha, V., Gera, R. & Soundarajan, S., Dec 1 2018, In: Social Network Analysis and Mining. 8, 1, 15.
Sampling 100%
data network 42%
intelligence 25%
illegal activities 20%
criminology 16%
Sampling design for Gaussian detection problems
Yu, C. T. & Varshney, P. K., Sep 1997, In: IEEE Transactions on Signal Processing. 45, 9, p. 2328-2337 10 p.
Yu, C. T. & Varshney, P. K., 1997, In: IEEE Transactions on Signal Processing. 45, 9, p. 23282337 1 p.
Sampling schemes for sequential detection in colored noise
Niu, R. & Varshney, P. K., Oct 7 2004, In: ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings. 2, p. II717-II720
Numerical methods 25%
Sampling schemes for sequential detection with dependent observations
Niu, R. & Varshney, P. K., Mar 1 2010, In: IEEE Transactions on Signal Processing. 58, 3 PART 2, p. 1469-1481 13 p.
Scalability of Inter-Domain Edge Tunnels Using Bandwidth Brokers: A Region-Based Approach
Okumus, I., Chapin, S. J., Hwangt, J. & Mantar, H., Dec 1 2003, p. 2903-2907. 5 p.
tunnel 100%
Tunnels 93%
Scalable graph-based bug search for firmware images
Feng, Q., Zhou, R., Xu, C., Cheng, Y., Testa, B. & Yin, H., Oct 24 2016, CCS 2016 - Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security. Association for Computing Machinery, p. 480-491 12 p. (Proceedings of the ACM Conference on Computer and Communications Security; vol. 24-28-October-2016).
Firmware 100%
Flow graphs 81%
Binary codes 25%
Scalable Log Auditing on Private Blockchains via Lightweight Log-Fork Prevention
Tang, Y., Li, K., Wang, Y. & Somuncuoglu, S. B., Dec 7 2020, SERIAL 2020 - Proceedings of the 2020 4th Workshop on Scalable and Resilient Infrastructures for Distributed Ledgers, Part of Middleware 2020. Association for Computing Machinery, Inc, p. 1-4 4 p. (SERIAL 2020 - Proceedings of the 2020 4th Workshop on Scalable and Resilient Infrastructures for Distributed Ledgers, Part of Middleware 2020).
Blockchain 100%
Servers 15%
Network protocols 14%
Scalable NoC-based Neuromorphic Hardware Learning and Inference
Fang, H., Shrestha, A., De Ma, M. & Qiu, Q., Oct 10 2018, 2018 International Joint Conference on Neural Networks, IJCNN 2018 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 8489619. (Proceedings of the International Joint Conference on Neural Networks; vol. 2018-July).
Network-on-chip 100%
Computer hardware 51%
Power and Energy 14%
Scale estimation with difference of ordered residuals
Scalzo-Cornacchia, M. & Velipasalar, S., Dec 9 2015, Proceedings - International Conference on Image Processing, ICIP. IEEE Computer Society, Vol. 2015-December. p. 3377-3381 5 p. 7351430
Computer vision 100%
Inspection 89%
Scattering Cross Section of Composite Conducting and Lossy Dielectric Bodies
Sarkar, T. K. & Arvas, E., May 1989, In: Proceedings of the IEEE. 77, 5, p. 788-795 8 p.
Scattering from arbitrary planar periodic screen consisting of resistive patches and dielectric layers
Petre, P., Sarkar, T. K., Zombory, L. & Swaminathan, M., Dec 1 1991, 1991 Digest. IEEE Computer Society, p. 1870-1873 4 p. (AP-S International Symposium (Digest) (IEEE Antennas and Propagation Society); vol. 3).
Fast Fourier transforms 100%
Scattering 98%
Green's function 61%
Scattering from perfectly conducting strips by utilizing an adaptive multiscale moment method – abstract
Su, C. & Sarkar, T. K., Jan 1 1998, In: Journal of Electromagnetic Waves and Applications. 12, 8, p. 1083-1084 2 p.
electromagnetic scattering 100%
strip 74%
moments 62%
Scattering from three-dimensional dispersive gyrotropic bodies using the TLM method
Yagli, A. F., Lee, J. K. & Arvas, E., 2009, In: Progress In Electromagnetics Research B. 18, p. 225-241 17 p.
Permittivity 100%
multistatic radar 94%
transmission lines 85%
radar cross sections 78%
SC-DCNN: Highly-scalable deep convolutional neural network using stochastic computing
Ren, A., Li, Z., Ding, C., Qiu, Q., Wang, Y., Li, J., Qian, X. & Yuan, B., Apr 4 2017, In: ACM SIGPLAN Notices. 52, 4, p. 405-418 14 p.
Ren, A., Li, Z., Ding, C., Qiu, Q., Wang, Y., Li, J., Qian, X. & Yuan, B., Apr 4 2017, ASPLOS 2017 - 22nd International Conference on Architectural Support for Programming Languages and Operating Systems. Association for Computing Machinery, p. 405-418 14 p. (International Conference on Architectural Support for Programming Languages and Operating Systems - ASPLOS; vol. Part F127193).
Scene recognition with audio-visual sensor fusion
Devicharan, D., Mehrotra, K. G., Mohan, C. K., Varshney, P. K. & Zuo, L., Nov 10 2005, In: Proceedings of SPIE - The International Society for Optical Engineering. 5813, p. 201-210 10 p., 22.
Sensor Fusion 100%
Audiovisual 64%
Vision 57%
Scheduling in D2D underlaid cellular networks with deadline constraints
Li, Y., Cenk Gursoy, M. & Velipasalar, S., Jul 2 2016, 2016 IEEE 84th Vehicular Technology Conference, VTC Fall 2016 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 7881211. (IEEE Vehicular Technology Conference; vol. 0).
Cellular Networks 100%
Deadline 90%
Scheduling Algorithm 49%
Scheduling algorithms 43%
Scheduling sleeping nodes in high density cluster-based sensor networks
Deng, J., Han, Y. S., Heinzelman, W. B. & Varshney, P. K., Dec 2005, In: Mobile Networks and Applications. 10, 6 SPEC. ISS., p. 825-835 11 p.
Sensor networks 100%
Sensor nodes 31%
Energy conservation 9%
Schelkunoff integrals for vertical dipoles
Dyab, W. M., Sarkar, T. K. & Salazar-Palma, M., Dec 1 2013, 2013 IEEE Antennas and Propagation Society International Symposium, APSURSI 2013 - Proceedings. p. 726-727 2 p. 6711022. (IEEE Antennas and Propagation Society, AP-S International Symposium (Digest)).
Score Computation for Linear Rank Procedures
Mehrotra, K. G. & White, T., Jan 1 1983, In: Journal of Statistical Computation and Simulation. 16, 3-4, p. 201-211 11 p.
SCUTA: A server-side access control system for web applications
Tan, X., Du, W., Luo, T. & Soundararaj, K. D., Jul 25 2012, SACMAT'12 - Proceedings of the 17th ACM Symposium on Access Control Models and Technologies. p. 71-82 12 p. (Proceedings of ACM Symposium on Access Control Models and Technologies, SACMAT).
Control systems 65%
Security of data 16%
Seamless Deductive Inference via Macros
Sahebolamri, A., Gilray, T. & Micinski, K., Sep 19 2022, CC 2022 - Proceedings of the 31st ACM SIGPLAN International Conference on Compiler Construction. Egger, B. & Smith, A. (eds.). Association for Computing Machinery, Inc, p. 77-88 12 p. (CC 2022 - Proceedings of the 31st ACM SIGPLAN International Conference on Compiler Construction).
Logic programming 100%
Computer programming languages 77%
Macros 76%
Static analysis 44%
Searching for better randomized response schemes for privacy-preserving data mining
Huang, Z., Du, W. & Teng, Z., 2007, Knowledge Discovery in Database: PKDD 2007 - 11th European Conference on Principles and Practice of Knowledge Discovery in Databases, Proceedings. Springer Verlag, p. 487-497 11 p. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); vol. 4702 LNAI).
Privacy Preserving Data Mining 100%
Randomized Response 66%
Data mining 41%
Nominal or categorical data 22%
Confidentiality 9%
Searching for high-value rare events with uncheatable grid computing
Du, W. & Goodrich, M. T., 2005, In: Lecture Notes in Computer Science. 3531, p. 122-137 16 p.
Grid Computing 100%
Rare Events 71%
Grid computing 66%
Supervisory personnel 65%
Coalitions 17%
Second-order plasma enabled tunable low-profile frequency selective surface based on coupling inter-layer
Payne, K., Peters, E. F., Brunett, J., Wedding, D. K., Wedding, C. A. & Choi, J. H., 2016, European Microwave Week 2016: "Microwaves Everywhere", EuMW 2016 - Conference Proceedings; 46th European Microwave Conference, EuMC 2016. Institute of Electrical and Electronics Engineers Inc., p. 309-312 4 p. 7824340. (European Microwave Week 2016: "Microwaves Everywhere", EuMW 2016 - Conference Proceedings; 46th European Microwave Conference, EuMC 2016).
Frequency selective surfaces 100%
selective surfaces 99%
Plasmas 71%
Switches 55%
Secrecy capacity of semi-deterministic wire-tap channels
Grubb, J., Vishwanath, S., Liang, Y. & Poor, H. V., Dec 1 2007, Proceedings of the 2007 IEEE Information Theory Workshop on Information Theory for Wireless Networks, ITW. p. 199-202 4 p. 4318063. (Proceedings of the 2007 IEEE Information Theory Workshop on Information Theory for Wireless Networks, ITW).
secrecy 100%
Wire 66%
Secrecy capacity region of binary and Gaussian multiple access channels
Liang, Y. & Vincent Poor, H., 2006, 44th Annual Allerton Conference on Communication, Control, and Computing 2006. University of Illinois at Urbana-Champaign, Coordinated Science Laboratory and Department of Computer and Electrical Engineering, p. 1098-1106 9 p. (44th Annual Allerton Conference on Communication, Control, and Computing 2006; vol. 3).
Secrecy capacity region of fading broadcast channels
Liang, Y., Poor, H. V. & Shamai, S., 2007, Proceedings - 2007 IEEE International Symposium on Information Theory, ISIT 2007. p. 1291-1295 5 p. 4557401. (IEEE International Symposium on Information Theory - Proceedings).
Broadcast Channel 100%
Fading 87%
Receiver 75%
Secrecy capacity region of parallel broadcast channels
Liang, Y., Poor, H. V. & Shamai, S., 2007, 2007 Information Theory and Applications Workshop, Conference Proceedings, ITA. p. 245-250 6 p. 4357587. (2007 Information Theory and Applications Workshop, Conference Proceedings, ITA).
Broadcast 94%
Node 37%
Information Needs 21%
Secrecy throughput of MANETs under passive and active attacks
Liang, Y., Poor, H. V. & Ying, L., Oct 2011, In: IEEE Transactions on Information Theory. 57, 10, p. 6692-6702 11 p., 6034725.
Mobile ad hoc networks 89%
Secure communication 19%
guarantee 12%
Secrecy throughput of MANETs with malicious nodes
Liang, Y., Poor, H. V. & Ying, L., Nov 19 2009, 2009 IEEE International Symposium on Information Theory, ISIT 2009. p. 1189-1193 5 p. 5205992. (IEEE International Symposium on Information Theory - Proceedings).
Mobile Ad Hoc Networks 100%
Vertex of a graph 30%
Critical Threshold 7%


T9 Items! Tier 7 Items free for all; [Hearthstone] - TAXI; All WotLK work! Battlegrounds & Arena working! Custom EVENTS! Super Donor Items; Vote reward! Enchant work,No LAG, NO Bugs.
Free TIER 10 and S8 - Over 1000 players - Always 100+ Online - Active Community - Working Spells - Well-Scripted Instances - Active BattleGrounds - MASSIVE CUSTOM CONTENTS - Custom INSTANCES - MASSIVE PVP - Working Vehicles and Dual Spec -Rewards.
Dedicated Server! Migration System! TrinityCore2 + TDB: WotLK Support! Scripted NR Instances(including Naxx), active pvp, great community!
24/7 Online stable connection server, great host. 100x all rates, level 80 max level cap. Custom made tier 9 armors, weapons and jewelry's. Custom bosses and custom events! Working Voting reward system! Client 3.2.2a Supported!!!
Bonjour, nous vous proposons un serveur wow en version 3.3.5a avec site web fonctionnel sécurisé, possibilités de parrainage, outils personnages, loterie et bien plus encore.


The number of people taking part in demonstrations matter. Social mobilisation is a crucial means to an end.
It seems to be all the rage at the moment to compare Barcelona to Glasgow. So I thought I'd continue that trend, and add my thoughts about social mobilisation in Catalonia and Scotland. To do this I've looked at a two events held in Barcelona last month. By looking at these events I will hopefully demonstrate my concern for the forthcoming second independence referendum in Scotland. I will set that thought out here: our events will have little impact if the current trend of low numbers continues.
Yup, there really was 50,000 outside the court on a Monday morning. This wasn't any kind of anomaly. Catalans demonstrate regularly and often bring tens of thousands on to the streets.
A week or so later an estimated 160,000 took to the streets on a Saturday evening to demonstrate against Spain's intransigence with refugees.
I wonder if Scottish politicians and the YES movement have realised that numbers really matter. And if so what they plan to do about it?


text:
5 Things: Stuart Wiggin on Filming Comedy in China
By Justine Lopez, July 26, 2016
A That's series where we ask a Beijing-based somebody to tell us 5 Things specific to his or her life.
You might know Stuart Wiggin from his work on China Plus' The C4 Show 别叫我憨豆 and its travel-themed offshoot Travelling with Stu-pid 司徒游记. The Beijing-based funny man is the writer and self-described "on-screen idiot" of the two comedy shows. We don't know about you, but we think he's hilarious. Most recently, Stuart has been on a comedy quest all across China from his most recent trip to Chongqing to the capital of Xinjiang, Urumqi. He's even joked with monks in the Qinghai Tibetan Autonomous Region and the Shaolin Temple.
So what's it really like to film a comedy show in China? Stuart gives us a behind-the-scenes look at 5 things he's learned about being a comedy star in China.
1. Some people will believe anything
The C4 Show films a mixture of comedy sketches and part scripted/part improvised travel shows. From my experience, comedy shows in China are usually 100 percent scripted, so in the case of our travel shows, we blur the lines a little. Obviously, whatever kind of video I'm filming, I'm portraying a character. The guy that people see on their screens, 司徒, is not necessarily the real me; although you'll have a hard time telling that to some of the people who follow the show.
Whether it's the fake phone calls home to my mom, pretending to urinate in the world's biggest public toilet, smashing up Tianjin's Porcelain House, running away from Hippies in Dali, or claiming to live as the Monkey King 24/7, as far as a large proportion of casual viewers are concerned, this all actually happened and the cameras just happened to be there.
This also extends to others who appear in the show. In one episode entitled "The Man Who Speaks Guitar," I conducted an (obviously fake) interview with a person who had been struck down by a disease called music mouth, which meant he was only able to communicate via his guitar (ridiculous, I know). After appearing on the show, his colleagues in real life approached him to ask whether it was true. To his credit, he kept it totally kayfabe and drummed his answer out on the table.
2. Just because you're famous doesn't mean you're famous
Even if you try really hard and get millions of viewers there's a pretty small chance that anyone will actually recognize you. Having been featured on the front pages of Youku, Tudou and Bilibili on a number of previous occasions, I have been recognized in the street a total of six times! That's a pretty pathetic payoff for three years of work.
3. Mundane things are usually the funniest
Some of our shows have plots, numerous filming location, beautiful B-roll, lots of jokes and maybe even a story arc. I lie, we never have a story arc. But some of our most popular videos have been about the most mundane of subjects. A parody/piss take on the ridiculously popular documentary A Bite of China remains one of our most successful shows. The subject: potatoes.
4. If you don't get it in one take, you just don't get it
If you happen to watch one of our comedy travel shows, you'll see that a lot of the stuff we film was done in one take. This obviously involves me being the wit (prick) that I am 100 percent of the time. It also involves my fabulous team (of two other people) taking advantage of every opportunity we get because the funniest things usually rely on spur-of-the-moment decisions and the people who just so happen to be around at that very moment.
5. Chinese culture is absolutely full of humor
Every aspect of Chinese culture can be funny. By this I don't mean that every aspect of Chinese culture can be made fun of and ridiculed; you can do that if you must but then you're probably just a racist. What I mean is that there's a funny side to everything, and Chinese viewers seem to appreciate it when that funny side is brought out. We have a series of sketches which explain every single Chinese holiday one by one, from Chinese Valentine's Day to Double 11 and even Tomb Sweeping Festival. Yes, we made a sketch about a festival which basically revolves around death. If you look hard enough, you can make everything funny.
If you want to watch the show, just search for it on the following platforms below:
Youku: "thec4show" or 别叫我憨豆
Bilibili: c4show
Weibo: @c4show
WeChat: c4show
Facebook: Comedy in China
Twitter: chinacomedy_c4
beijing 5 things stuary wiggin comedy television comedy comedy show
We reached out to Liu to learn how she got Banana Tour off the ground and get some travel recommendations.
This Day in History: Anna May Wong's Shanghai Express
The first Chinese-American movie star's struggle for acceptance in the land of her ancestors.
This Day in History: Google Announces Exit from China Market
Google's China-based google.cn search webpage, launched in 2006, had a turbulent time operating on the Chinese mainland.
Four Places For Outdoor Ice Skating in Beijing
It's that time of year when Beijing's lakes and rivers are frozen over, the perfect time to get your skates on. Here are four suggestions for outdoor ice skating in Beijing.
Explainer: Why Winter is the Most Polluted Time of Year in China
'Tis the season for pollution and facemasks.
Where to Ski in Beijing Without Breaking The Bank
Shelling out on a long ski excursion to Xinjiang, Chongli or Changbai Mountain may not be an option for everyone. So for those living in or paying a visit to Beijing, here are five skiing trips which won't break the bank.
Explainer: Why China Celebrates Christmas with Apples
Ever wonder why you've been receiving apples from Chinese friends and co-workers for Christmas?
Exploring Shanghai's Largest Freshwater Lake
Find out whether Dianshan Lake is a not-so-hidden gem or just another cluster of construction sites.
Sign up to get the entertainment, lifestyle and event news from That's Beijing every week!
China Records First Coronavirus Death Since May as Cases Surge
Traveling Over Spring Festival Might Not Be So Easy
These are China's Most 'Digitalized' Cities
The Benefits of Biological Diversity in the City
New Menus and Openings in Beijing
We Tried McDonald's Roujiamo & They Should Leave It to the Pros
Beijing Changes Quarantine Policy... Again
Ice & Snow Sculpture Festival and a Whole Lot More in Harbin
Scan our QR Code at right or follow us at ThatsBeijing for events, guides, giveaways and much more!
7 Days in Beijing With thatsmags.com
Never miss an issue of That's Beijing!