text:
\section{Conclusion}
\label{chap:conclusion}

We introduce a novel family of methods for neural network pruning based on Gibbs distributions that achieves high pruning rates with little reduction in accuracy. It can be used for either unstructured or structured pruning, and the general framework can be adapted to a wide range of pruning structures. We compare it to various established pruning methods and find that it outperforms them. The success of our proposed methods shows the efficacy of simultaneously training and pruning a network and using stochasticity to promote robustness under pruning.

Future work could explore other types of structured pruning that can be expressed within this framework, such as limiting the number of non-zero values in convolutional kernels or structured pruning of recurrent neural network layers. Results could also be refined by determining an optimal pruning rate for each layer given a desired global pruning rate, rather than just using the same rate for each layer. Gibbs pruning could also be combined with other network compression methods like quantization to obtain highly efficient neural networks in practice.

\section{Evaluation}
\label{chap:evaluation}

We compare Gibbs pruning to several other structured and unstructured pruning methods. The other methods were chosen to represent current practices and research, either being
common, well-established pruning method or ones that have recently shown exceptional results. The lack of standard experimental setups in network pruning make it difficult to judge which methods are state-of-the-art~\cite{blalock2020state}, especially in the case of structured pruning, but we do compare to best known results where possible.

We evaluate performance on ResNet neural networks~\cite{he2016deep}, which is a common approach in the network pruning literature~\cite{blalock2020state}. Nearly all of the parameters of these networks are in convolutional layers, which already are much more sparse than dense layers of similar dimensions and additionally employ weight sharing, making them more difficult to prune than dense layers. Pruning methods that are effective on networks like AlexNet~\cite{krizhevsky2012imagenet} with large dense layers often do not show the same performance on networks like ResNet, making them a more challenging test.

In particular, we train ResNet-20 and ResNet-56 with linear projection~\cite{he2016deep}. ResNet-56 was chosen because it is the most common ResNet variant used in evaluating pruning methods~\cite{blalock2020state}, making it easier to compare our method to others. ResNet-20 on the other hand is the smallest standard ResNet variant. This makes it a useful test of pruning methods because it is less likely to be overparameterized for particular datasets, and it can be used with high pruning rates to investigate the efficacy of extremely small neural networks.

We run our experiments with the CIFAR-10 dataset~\cite{krizhevsky2012cifar}. This is a standard image classification dataset that is widely used in evaluating pruning methods~\cite{blalock2020state}.

The networks are trained for 200 epochs using the Adam optimizer~\cite{kingma2014adam}, with a learning rate initially set to \(10^{-3}\) and reduced by a factor of 10 at epoch 80 and every 40 epochs thereafter. We use data augmentation during training as in~\cite{he2016deep}, randomly shifting images horizontally and vertically by up to 10\% and flipping images horizontally with 50\% probability. This achieves baseline top-1 accuracies of 90.7\% for ResNet-20 and 92.2\% for ResNet-56.

For all methods, we prune all convolutional layers except for the first one, following the recommendation in \cite{gale2019state}. This is because the input to the first convolutional layer only has three channels, leading to the first convolutional layer having very few parameters and having its performance easily harmed by high pruning rates. A precise summary of pruned and unpruned layers is shown in Table \ref{t:params}.

\begin{table}[t!]
	\centering
	\caption{The fractions of parameters in pruned and unpruned layers in the evaluated neural networks.}
	\begin{tabular}{cc|c|c}
		\multicolumn{2}{c}{} & ResNet-20 & ResNet-56 \\\hline
		\multirowcell{3}{\textbf{Unpruned} \\ \textbf{layers}} & First convolution & 0.16\% & 0.05\% \\
										 & Batch normalization & 1.01\% & 0.95\% \\
										 & Dense & 0.24\% & 0.08\%  \\\hline
		\makecell{\textbf{Pruned} \\ \textbf{layers}} & Other convolutions & 98.60\% & 98.93\% \\
	\end{tabular}
	\label{t:params}
\end{table}

For pruning methods that use additional epochs for fine-tuning, we use a training rate of \(10^{-5}\) after the initial training phase. We also test changing the lengths of fine-tuning schedules, to evaluate tradeoffs between the number of additional training epochs and the final accuracy.

\subsection{Unstructured Pruning Results}
\label{sec:ev-unstruct}

The Hamiltonians previously discussed for unstructured pruning are the binary Hamiltonian (\ref{eq:bin}) and the linear Hamiltonians given by (\ref{eq:a1}), (\ref{eq:a2}), and (\ref{eq:a3}). Initial and final \(\beta\) values, as well as the number of epochs to anneal over and the use of a logarithmic or linear schedule were manually tuned to maximize final accuracy. The best results achieved for each are shown in Table \ref{t:unstruct-hams}. Since the best result was observed with (\ref{eq:a2}), we use this formulation in the rest of our experiments.

\begin{table}[t!]
	\centering
	\caption[Comparison of Hamiltonians for unstructured pruning.]{Comparison of Hamiltonians for unstructured pruning. 90\% of weights in each pruned layer are pruned.}
	\label{t:results_unstruct}
	\begin{tabular}{cc|cc}
		Variant & \makecell{ResNet-20\\Accuracy} & \makecell{ResNet-56\\Accuracy} \\\hline
		Binary (\ref{eq:bin}) & 86.6 & 89.1 \\
		Linear (\ref{eq:a1}) & 85.7 & 87.7 \\
		Linear (\ref{eq:a2}) & \textbf{87.5} & \textbf{89.8} \\
		Linear (\ref{eq:a3}) & 86.9 & 89.2
	\end{tabular}
	\label{t:unstruct-hams}
\end{table}

Notably, (\ref{eq:a2}) and (\ref{eq:a3}), which produced the best results, both create a Hamiltonian with coefficients that don't just express whether a weight should be pruned or not, but also vary depending on how far a weight is from \(Q(p,{\mathbf w})\). This allows weights with very high or very low magnitudes to converge to a pruned or unpruned state earlier. This being desirable would also explain why (\ref{eq:a2}) is more effective than (\ref{eq:a3}), since it considers the squares of magnitude, making weights with high magnitudes vary from \(Q(p,{\mathbf w})\) even more and therefore converge earlier.

After manually tuning \(\beta\) annealing for the Hamiltonian (\ref{eq:a2}), we arrived at a logarithmic schedule from 0.7 to 10000 done over the first 128 epochs of a 200 epoch schedule, updating \(\beta\) at the end of each epoch. This schedule is used for all subsequent unstructured pruning experiments.
In practice, we find that when using schedule stretching with very long schedules, the pruning mask does not always converge at \(\beta=10^4\). This indicates that the empirical distribution of weights can be significantly different in some highly stretched configurations. In these cases we increase the final beta to \(10^6\), which achieves convergence in all such cases.

\subsubsection{Comparison to Existing Methods}

We compare Gibbs pruning to four established unstructured pruning methods. For methods that use fine-tuning, we evaluate spending different amounts of time on fine-tuning. We also test different overall training times for Gibbs pruning by using schedule stretching as described in Section \ref{sec:gp-schedule}.

The first established method is using \(l_1\) regularization. We use a penalty of 0.001, which was manually tuned to maximize final accuracy. After training, we mask the fraction \(p\) of weights in each layer with the lowest magnitude. \(l_1\) regularization is known to cause weights to become sparse~\cite{goodfellow2016deep}, and has been used as a simple point of comparison for unstructured pruning methods~\cite{gomez2019learning}.

The other pruning methods we compare to are more advanced approaches from the literature. The first of these is the method proposed in \cite{han2015learning} by Han et al. Once training is complete, a certain percentage of the weights with lowest magnitude are pruned and the network is fine-tuned for some time. This procedure is then repeated several times. We test different schedule lengths in order to compare this methods to other methods that can use different training schedule lengths. Before each fine-tuning period, we increase the pruning rate by 10\% up to 90\%, fine-tuning for \(n\) epochs each time, leading to a total of \(200+9n\) epochs.

We also compare to iterative magnitude pruning (IMP), proposed by Frankle et al.~\cite{frankle2019lottery} with rewinding~\cite{frankle2019stabilizing}. This method trains the network several times, pruning gradually more each time and then rewinding the network weights to the values they had after the first 500 training steps. To test different training times, we vary the number of times that the network is trained with intermediate pruning rates between the initial rate of 0\% and the final rate of 90\%.

Finally, we test targeted dropout, a method recently proposed by Gomez et al.\cite{gomez2019learning}. In this method, the fraction \(\gamma\) of weights with lowest magnitude are each omitted from the network with probability \(\alpha\) at each training step. After training, the fraction of weights \(p\) with lowest magnitude are permanently pruned.
We use the authors' most successful hyperparameter settings: \(\alpha=0.75,\gamma=0.9\). We also evaluate ramping targeted dropout, a variant where \(\alpha\) is gradually changed from 0 to \(p\) during training~\cite{gomez2019learning}.

\begin{figure}[t!]
	\centering
	\begin{subfigure}{0.48\textwidth}
	\includegraphics[width=\textwidth]{journal-resnet20-scatter.pdf}
	\label{fig:resnet20scatter}
	\vspace{-7mm}
	\caption{ResNet-20}
	\end{subfigure}
	\begin{subfigure}{0.48\textwidth}
	\includegraphics[width=\textwidth]{journal-resnet56-scatter.pdf}
	\label{fig:resnet56scatter}
	\vspace{-7mm}
	\caption{ResNet-56}
	\end{subfigure}
	\caption[Comparison of Gibbs pruning to other unstructured pruning methods.]{Comparison of Gibbs pruning to other unstructured pruning methods. Dotted lines are shown for comparison on methods that do not require additional training epochs. }
	\label{fig:resnetscatter}
\end{figure}

\begin{figure}[t!]
	\centering
	\includegraphics[width=0.48\textwidth]{journal-p-acc-plot.pdf}
	\caption{Loss in accuracy compared to the baseline versus pruning rate using unstructured Gibbs pruning.}
	\label{fig:p-acc}
\end{figure}

We evaluate pruning 90\% of weights in each pruned layer, which is a high degree of compression for the given networks. Because this pruning rate is challenging, it effectively shows performance differences between different methods. Results are shown in Figure~\ref{fig:resnetscatter}. These show that Gibbs pruning is effective without additional training epochs, giving similar results to the best results from the other evaluated methods. When the training time is extended using schedule stretching, Gibbs pruning decisively outperforms the other evaluated methods.

\subsubsection{High Pruning Rates}

We evaluate Gibbs pruning using different pruning rates in Figure~\ref{fig:p-acc}. When schedule stretching is used, the resulting accuracy can come close to or exceed the baseline accuracy even with very high pruning rates. For instance, removing 95\% of weights in pruned layers for an overall pruning rate of 94\% decreases accuracy by only 0.90\% for ResNet-20 and 0.45\% for ResNet-50. This corresponds to models with roughly 17\,000 final parameters when pruning ResNet-20 and 48\,000 final parameters when pruning ResNet-56. To the best of our knowledge, these are the best results reported for such a high compression rate on CIFAR-10~\cite{blalock2020state}.

\subsubsection{Random Masks and Re-initialization}
\label{sec:ev-re-initialization}

\begin{table}[t!]
	\centering
	\caption[Comparison of random masks and random weight re-initialization to Gibbs pruning.]{Comparison of random masks and random weight re-initialization to Gibbs pruning. Values are top-1 accuracy in percent.}
	\begin{subtable}{0.48\textwidth}
		\centering
		\caption{ResNet-20}
		\begin{tabular}{c|cccc}
			Method & \multicolumn{4}{c}{Pruning rate} \\\hline
				   & 50\% & 75\% & 90\% & 95\% \\\hline
				   Random mask & 89.3 & 87.2 & 83.2 & 79.8 \\
				   Random re-init. & 88.6 & 86.3 & 83.4 & 79.4 \\
				   Gibbs pruning & \textbf{89.9} & \textbf{89.0} & \textbf{87.5} & \textbf{85.2} \\
			   \end{tabular}
			   \vspace{2mm}
		   \end{subtable} \\
		   \begin{subtable}{0.48\textwidth}
			   \centering
			   \caption{ResNet-56}
			   \begin{tabular}{c|cccc}
				   Method & \multicolumn{4}{c}{Pruning rate} \\\hline
						  & 50\% & 75\% & 90\% & 95\% \\\hline
						   Random mask & \textbf{91.2} & 90.0 & 87.3 & 84.6 \\
						  Random re-init. & 90.3 & 89.1 & 86.7 & 84.5 \\
						  Gibbs pruning & \textbf{91.2} & \textbf{90.8} & \textbf{89.8} & \textbf{88.5} \\
					  \end{tabular}
				  \end{subtable}
				  \label{t:random}
	   \end{table}

As previously discussed, many pruning methods do not produce a higher accuracy than if the pruned network is re-initialized and retrained~\cite{liu2019rethinking}, calling into question the utility of training large networks and then pruning. To assess this phenomenon with Gibbs pruning, we evaluate retraining pruned networks with randomly re-initialized weights. For comparison, we also train networks from scratch using random pruning masks with the chosen sparsity.

Results are shown in Table \ref{t:random}. We find that random re-initialization does not meet the performance of our proposed method, and is more similar to training with a random mask. These results suggest that Gibbs pruning acts less as a search over possible network architectures, and instead adapts more dynamically along with weight values throughout training, resulting in a final representation that is more tuned to particular weights than other methods are.

\subsection{Structured Pruning Results}
\label{sec:ev-struct}

We also evaluate performance for structured convolutional neural network pruning. We consider pruning individual kernels as well as entire filters of size \(K\times K\times C\), where \(K\) is the kernel width and \(C\) is the number of input channels for the layer being pruned. Generally speaking, kernel-wise pruning can achieve higher accuracies for the same number of parameters, but might be harder to optimize performance for, compared to filter-wise pruning.

We compare the quadratic Hamiltonian (\ref{eq:struct_ham}) to the other Hamiltonians we proposed for structured pruning: the binary Hamiltonian (\ref{eq:bin}) and the linear Hamiltonian (\ref{eq:struct_ham_lin}). Results are shown in Table \ref{t:results_structured}. For both cases, the quadratic Hamiltonian produced the best result, so we use it for all other experiments.

In practice, we find that moving to kernel- or filter-wise pruning with the chosen Hamiltonians does not significantly affect the convergence of pruning masks as a function of \(\beta\). We therefore use the same \(\beta\) schedule as in unstructured pruning, annealing from 0.7 to 10\,000 over the first 128 epochs. Given this schedule, we manually tune \(c\). We find that a value around \(0.01\) produces the best final results, so this value is used for all subsequent experiments.

\begin{table}[t!]
	\centering
	\caption{Comparison of Hamiltonians for structured pruning.}
	\label{t:results_structured}
	\vspace{-2mm}
	\begin{subtable}{0.48\textwidth}
		\centering
		\caption{Kernel-wise pruning}
		\begin{tabular}{c|cc}
			Hamiltonian & \makecell{ResNet-20\\Accuracy} & \makecell{ResNet-56\\Accuracy} \\\hline
			Binary (\ref{eq:bin}) & 83.5 & 87.7 \\
			Linear (\ref{eq:struct_ham_lin}) & 82.0 & 86.5 \\
			Quadratic (\ref{eq:struct_ham}) & \textbf{84.8} & \textbf{88.2}
		\end{tabular}
		\vspace{2mm}
	\end{subtable} \\
	\begin{subtable}{0.48\textwidth}
		\centering
		\caption{Filter-wise pruning}
		\begin{tabular}{c|cc}
			Hamiltonian & \makecell{ResNet-20\\Accuracy} & \makecell{ResNet-56\\Accuracy} \\\hline
			Binary (\ref{eq:bin}) & 82.3 & 87.0 \\
			Linear (\ref{eq:struct_ham_lin}) & 79.7 & 86.3 \\
			Quadratic (\ref{eq:struct_ham}) & \textbf{83.5} & \textbf{87.5}
		\end{tabular}
	\end{subtable}
\end{table}

For kernel-wise Gibbs pruning, each neighbourhood is small enough that the partition function is practically computable, so we sample directly from the Gibbs distribution. For filter-wise pruning, we use chromatic Gibbs sampling as described in Section~\ref{sec:struct-sampling}. We run the Markov chain for 50 iterations, as further iterations were not observed to affect performance in practice.

\begin{figure}[t!]
	\centering
	\begin{subfigure}{0.48\textwidth}
	\includegraphics[width=\textwidth]{journal-resnet20-kernel-scatter.pdf}
	\label{fig:resnet20kernelscatter}
	\vspace{-7mm}
	\caption{ResNet-20}
	\end{subfigure}

	\begin{subfigure}{0.48\textwidth}
	\includegraphics[width=\textwidth]{journal-resnet56-kernel-scatter.pdf}
	\label{fig:resnet56kernelscatter}
	\vspace{-7mm}
	\caption{ResNet-56}
	\end{subfigure}
	\caption[Comparison of Gibbs pruning to other kernel-wise structured pruning methods.]{Comparison of Gibbs pruning to other kernel-wise structured pruning methods. Dotted lines are shown for comparison on methods that do not require additional training epochs. }
	\label{fig:resnetkernelscatter}
\end{figure}

\begin{figure}[t!]
	\centering
	\begin{subfigure}{0.48\textwidth}
	\includegraphics[width=\textwidth]{journal-resnet20-filter-scatter.pdf}
	\label{fig:resnet20filterscatter}
	\vspace{-7mm}
	\caption{ResNet-20}
	\end{subfigure}

	\begin{subfigure}{0.48\textwidth}
	\includegraphics[width=\textwidth]{journal-resnet56-filter-scatter.pdf}
	\label{fig:resnet56filterscatter}
	\vspace{-7mm}
	\caption{ResNet-56}
	\end{subfigure}
	\caption[Comparison of Gibbs pruning to other filter-wise structured pruning methods.]{Comparison of Gibbs pruning to other filter-wise structured pruning methods. Dotted lines are shown for comparison on methods that do not require additional training epochs. }
	\label{fig:resnetfilterscatter}
\end{figure}

We compare Gibbs pruning to three established structured pruning methods. One is targeted dropout~\cite{gomez2019learning}, implemented as previously described, but dropping entire kernels or filters. For filter-wise pruning, we set its hyperparameters to the most effective values described for unit dropout: \(\alpha=0.9, \gamma=0.75\). We also test the ramping targeted dropout variant as previously described.

Another method we evaluate is iteratively pruning filters or kernels based on their \(l_1\)-norm and retraining, as proposed by Li et al. in \cite{li2017pruning}. We use the same iterative schedules of pruning and fine-tuning as previously described for \cite{han2015learning} to test different total training lengths.

Finally, we evaluate ThiNet pruning, as proposed by Luo et al.~\cite{luo2017thinet}. This method greedily prunes channels to minimize changes in activations over an evaluation dataset. Pruning a channel is equivalent to pruning a filter in the previous layer, making channel-wise and filter-wise pruning comparable. ThiNet does not prune the \(1\times1\) convolution layers used for linear projection, which we also omit for filter-wise Gibbs pruning for comparison.

Kernel-wise pruning results are shown in Figure~\ref{fig:resnetkernelscatter} and filter-wise pruning results are shown in Figure~\ref{fig:resnetfilterscatter}. Note that a lower pruning rate of 75\% is used in evaluating filter-wise pruning methods, since filter-wise pruning at high rates results in much lower accuracy. In both types of structured pruning, Gibbs pruning outperforms other methods, and stretching the learning rate and annealing schedules proves to be effective in additionally improving accuracy.

\subsection{Analysis}
\label{sec:ev-summary}

After evaluating our proposed methods, we find that we obtain the best results with the Hamiltonians given by (\ref{eq:a2}) for unstructured pruning and (\ref{eq:struct_ham}) for structured pruning. We compare to a number of existing pruning methods and find that Gibbs pruning outperforms them. In particular, we establish a new state-of-the-art result for unstructured pruning on CIFAR-10 with ResNet-56. We also demonstrate that, unlike the methods evaluated in \cite{liu2019rethinking}, Gibbs pruning outperforms a network trained with the same pruning mask but with random weights, showing that it effectively uses network weight values in selecting pruning masks.

The following reasons could explain why our proposed method outperforms existing methods. First of all, many previous works did not test their proposed methods on sufficiently challenging tasks, either using neural networks with large dense layers or using relatively low pruning rates, meaning that their actual efficacy compared to other state-of-the-art methods was unclear. Most existing methods also do not co-adapt weights and pruning masks during training, meaning that they do not find architectures that are particularly well-adapted to the parameter values in the network, as shown in~\cite{liu2019rethinking}. Finally, our novel uses of stochastic regularization for pruning and schedule stretching are significant changes from established approaches, and they have proven especially effective.

\section{Gibbs Pruning}
\label{chap:theory}

Gibbs pruning is a layer-wise pruning method, meaning that every layer in a neural network is pruned independently. Note that we do not prune regularization or pooling layers, and we may omit other layers from pruning if they have very few weights and pruning them will significantly harm performance. Since Gibbs pruning acts on individual layers, we describe it in terms of a single layer. Let \({\mathbf w} \in {\mathbb R}^N\) represent the layer's weights, flattened into a single vector in an arbitrary order. Let \({\mathbf x} \in \{-1,1\}^N\) represent a \textbf{pruning mask} for the layer, where \(x_i=-1\) means that \(w_i\) is pruned and \(x_i=1\) means that \(w_i\) is not pruned. We always use \(N\) to denote the number of weights in a layer.

In Gibbs pruning, \({\mathbf x}\) is sampled from a \textbf{Gibbs distribution} \(p({\mathbf x})\) at every training step and the layer weights are either retained or masked out during the forward pass based on the corresponding values in \({\mathbf x}\). The distribution is sampled once again after training to determine the final pruning mask. Note that weight values are not permanently modified during training, allowing a connection to be masked out in some iterations and active in others. Per-iteration mask sampling is illustrated in Figure \ref{fig:prob-pruning}.

A Gibbs distribution over \({\mathbf x}\) has the probability mass function:
\begin{equation}
	p({\mathbf x}) = \frac{1}{Z(\beta)}e^{-\beta H({\mathbf x})},
	\label{eq:gibbs}
\end{equation}
where \(\beta \in (0,\infty)\) is an inverse temperature parameter, which controls the overall tendency of the distribution to take a low-energy state, and \(Z(\beta)\) is a partition function, which is set to ensure that the probability mass function sums to 1.

Many applications of Gibbs distributions gradually increase \(\beta\) over many sampling iterations to move the distribution from a disordered state to an ordered state. This procedure is called \textbf{annealing}, by analogy to annealing heat treatments in metallurgy. The value of \(\beta\) is inversely proportional to a simulated system temperature, so annealing can be understood as gradually reducing the temperature of the system over time. Using annealing requires defining a schedule for changing \(\beta\) over time, which we call the \textbf{annealing schedule}.

The main idea behind our proposed method is to gradually converge to a pruning mask during the course of an ordinary neural network training procedure. This allows the weights of the neural network and the pruning mask to dynamically adapt to one another. We hypothesize that this approach is more effective than either training a smaller network or pruning a network that is already trained, even if fine-tuning is used. In particular, we want to train the network weights in such a way that they are resilient to pruning, which is something that is not generally done in pruning methods that prune trained networks and then perform fine-tuning.

Gibbs distributions are a natural fit for this kind of pruning. They permit annealing, which allows for gradual convergence from unpredictable behaviour to a final steady state in a controllable way. They are also a very general family of probability distributions, giving considerable flexibility in defining pruning methods. For instance, every Markov random field corresponds to a Gibbs distribution, and vice versa~\cite{geman1984stochastic}.

\begin{figure}[t!]
	\centering
	\begin{adjustbox}{width=0.42\textwidth}
	\begin{tikzpicture}[scale=1.0]
		\node[circle,minimum size=20,draw,fill=purple](1) at (0,0) {};
		\node[circle,minimum size=20,draw,fill=purple](2) at (0,1.33) {};
		\node[circle,minimum size=20,draw,fill=purple](3) at (0,2.66) {};
		\node[rectangle,minimum width=0.75cm,minimum height=0.5cm,draw,fill=white](s1)  at (2,-0.67) {\small +1};
		\node[rectangle,minimum width=0.75cm,minimum height=0.5cm,draw,fill=black,text=white](s2)  at (2,-0.17) {\small -1};
		\node[rectangle,minimum width=0.75cm,minimum height=0.5cm,draw,fill=white](s3)  at (2,0.33) {\small +1};
		\node[rectangle,minimum width=0.75cm,minimum height=0.5cm,draw,fill=white](s4)  at (2,0.83) {\small +1};
		\node[rectangle,minimum width=0.75cm,minimum height=0.5cm,draw,fill=white](s5)  at (2,1.33) {\small +1};
		\node[rectangle,minimum width=0.75cm,minimum height=0.5cm,draw,fill=black,text=white](s6)  at (2,1.83) {\small -1};
		\node[rectangle,minimum width=0.75cm,minimum height=0.5cm,draw,fill=black,text=white](s7)  at (2,2.33) {\small -1};
		\node[rectangle,minimum width=0.75cm,minimum height=0.5cm,draw,fill=black,text=white](s8)  at (2,2.83) {\small -1};
		\node[rectangle,minimum width=0.75cm,minimum height=0.5cm,draw,fill=white](s9)  at (2,3.33) {\small +1};
		\draw [-Stealth] (1) -- (s1.west); \draw [-Stealth] (1) -- (s4.west); \draw [-Stealth] (1) -- (s7.west);
		\draw [-Stealth] (2) -- (s2.west); \draw [-Stealth] (2) -- (s5.west); \draw [-Stealth] (2) -- (s8.west);
		\draw [-Stealth] (3) -- (s3.west); \draw [-Stealth] (3) -- (s6.west); \draw [-Stealth] (3) -- (s9.west);
		\node[circle,minimum size=20,draw,fill=purple](6) at (4,0) {};
		\node[circle,minimum size=20,draw,fill=purple](7) at (4,1.33) {};
		\node[circle,minimum size=20,draw,fill=purple](8) at (4,2.66) {};
		\draw [-Stealth] (s1.east) -- (6); \draw [-Stealth] (s3.east) -- (6);
		\draw [-Stealth] (s4.east) -- (7); \draw [-Stealth] (s5.east) -- (7);
		\draw [-Stealth] (s9.east) -- (8);
		\node[rectangle,minimum width=0.75cm,minimum height=0.5cm,draw,fill=black,text=white](s10)  at (6,-0.67) {-1};
		\node[rectangle,minimum width=0.75cm,minimum height=0.5cm,draw,fill=black,text=white](s11)  at (6,-0.17) {-1};
		\node[rectangle,minimum width=0.75cm,minimum height=0.5cm,draw,fill=white](s12)  at (6,0.33) {+1};
		\node[rectangle,minimum width=0.75cm,minimum height=0.5cm,draw,fill=black,text=white](s13)  at (6,0.83) {-1};
		\node[rectangle,minimum width=0.75cm,minimum height=0.5cm,draw,fill=white](s14)  at (6,1.33) {+1};
		\node[rectangle,minimum width=0.75cm,minimum height=0.5cm,draw,fill=white](s15)  at (6,1.83) {+1};
		\node[rectangle,minimum width=0.75cm,minimum height=0.5cm,draw,fill=white](s16)  at (6,2.33) {-1};
		\node[rectangle,minimum width=0.75cm,minimum height=0.5cm,draw,fill=black,text=white](s17)  at (6,2.83) {-1};
		\node[rectangle,minimum width=0.75cm,minimum height=0.5cm,draw,fill=black,text=white](s18)  at (6,3.33) {-1};
		\draw [-Stealth] (6) -- (s10.west); \draw [-Stealth] (6) -- (s13.west); \draw [-Stealth] (6) -- (s16.west);
		\draw [-Stealth] (7) -- (s11.west); \draw [-Stealth] (7) -- (s14.west); \draw [-Stealth] (7) -- (s17.west);
		\draw [-Stealth] (8) -- (s12.west); \draw [-Stealth] (8) -- (s15.west); \draw [-Stealth] (8) -- (s18.west);
		\node[circle,minimum size=20,draw,fill=purple](9) at  (8,0) {};
		\node[circle,minimum size=20,draw,fill=purple](10) at (8,1.33) {};
		\node[circle,minimum size=20,draw,fill=purple](11) at (8,2.66) {};
		\draw [-Stealth] (s12.east) -- (9);
		\draw [-Stealth] (s14.east) -- (10); \draw [-Stealth] (s15.east) -- (10);
		\draw [-Stealth] (s16.east) -- (11);

		\node[] at (2,-1.5) {\({\mathbf x}^{(1)} \sim p({\mathbf x}^{(1)})\)};
		\node[] at (6,-1.5) {\({\mathbf x}^{(2)} \sim p({\mathbf x}^{(2)})\)};
	\end{tikzpicture}
	\end{adjustbox}
	\caption{Connection pruning in a neural network using probability distributions over pruning masks.}
	\label{fig:prob-pruning}
\end{figure}

This high-level description provides a framework in which a wide range of pruning methods could be expressed.
In order to define an exact method, three elements need to be determined:
\begin{itemize}
	\item The Hamiltonian \(H({\mathbf x})\);
	\item An approach for annealing \(\beta\) during training;
	\item A method for sampling from the distribution.
\end{itemize}

\subsection{Properties of Gibbs Distributions}
\label{sec:gp-gibbs-properties}

As previously stated, Gibbs distributions can be annealed from behaving unpredictably to converging to a known final state. To be precise, at a high temperature or as \(\beta \rightarrow 0\), the distribution is effectively uniform over all \({\mathbf x}\), and at a low temperature or as \(\beta \rightarrow \infty\), samples from the distribution converge to only those \({\mathbf x}\) values that minimize \(H({\mathbf x})\)~\cite{geman1984stochastic}.

In order to define the Hamiltonian, we consider what the final converged state of the pruning mask should be. We can use some heuristic to determine a pruning mask that is likely be effective and set our Hamiltonian so that this state minimizes \(H({\mathbf x})\), which allows us to ensure that the annealing procedure will arrive at that state as \(\beta \rightarrow \infty\).

Aside from this constraint, the Hamiltonian could take any form, and could incorporate information like the current network parameters, the input data or intermediate values in the network, the initial network parameters, etc. While past pruning methods have considered a range of similar factors, we follow a number of contemporary works that find that simply looking at current weight magnitudes is enough to effectively prune networks~\cite{gale2019state,frankle2019lottery,han2016deep,guo2016dynamic}, and can outperform more complex metrics~\cite{gale2019state}. So, our Hamiltonians are only functions of the current weight magnitudes, the mask \({\mathbf x}\), and some hyperparameters that are held constant. In Sections \ref{sec:gp-unstruct} and \ref{sec:gp-struct}, we explore possible specific functions to use for \(H({\mathbf x})\), and we test their efficacies experimentally in Section \ref{chap:evaluation}.

We perform annealing by increasing \(\beta\) from a low value to a high value while training. At high temperatures (low \(\beta\)), \({\mathbf x}\) is roughly uniformly distributed over \(\{-1,1\}^N\), which acts similarly to stochastic regularization methods and in particular dropconnect~\cite{wan2013regularization}. This phase regularizes the network and conditions it to be robust under weight pruning. Once annealed to a low temperature (high \(\beta\)), the Gibbs distribution converges towards our desired final pruning mask. A significant amount of later training time can therefore be spent adapting to the particular structure of the pruning mask. The balance between time spent on adapting to random masks and time spent on adapting to the final mask can be controlled by the particular annealing schedule for \(\beta\) used in training.

In general, Gibbs distributions are difficult to sample from. This is because the partition function \(Z(\beta)\) is a sum containing a term for each possible value of \({\mathbf x}\), and therefore can grow exponentially in complexity in \(N\). However, for some Hamiltonians, the distribution simplifies in ways that enable easy sampling. For others, Markov chain Monte Carlo (MCMC) methods that allow sampling without explicitly calculating the partition function must be used. We propose sampling methods for each Hamiltonian we use in the following sections.

Another useful property of Gibbs distributions is that a Hamiltonian \(H({\mathbf x})=f({\mathbf x})+\gamma\) with some constant \(\gamma\in{\mathbb R}\) produces the same distribution as the Hamiltonian \(H({\mathbf x})=f({\mathbf x})\). This property holds because \(e^{\beta H({\mathbf x})}\) turns the constant additive factor to a constant multiplicative factor, which is then cancelled out by the partition function.

\subsection{Definitions}
\label{sec:gp-defns}

\(p\in[0,1]\) represents the pruning fraction, that is the fraction of weights that are removed in the final pruned layer.
For theoretical analysis, we require that \(p\) represent a pruning fraction that is achievable given the number of pruned weights or structures in the layer. For instance, if a layer contained four filters, pruning 80\% of filters (\(p=0.8\)) would be impossible. In practice, the number of weights is often large enough that achieving a pruning fraction of \(p\) can be closely approximated without having to exactly meet it.

We define an operation \(Q(p,{\mathbf w})\) as follows. Let \({\mathbf v}\) represent a vector with the elements of \({\mathbf w}\) sorted by magnitude such that \(v_i^2 \leq v_{i+1}^2\). Given a pruning fraction \(p\) between 0 and 1, select \(Q(p,{\mathbf w}) = v_i^2\), where \(i = p(N-1)+1\). If \(i\) is not an integer for the given value of \(p\), we instead linearly interpolate between the two nearest defined points \(Q(p^-,{\mathbf w})\) and \(Q(p^+,{\mathbf w})\):
\begin{gather}
	\text{Let } p^- = \left\lfloor\frac{i-1}{N-1}\right\rfloor,\quad p^+ = \left\lceil\frac{i-1}{N-1}\right\rceil, \nonumber \\
    Q(p,{\mathbf w}) = Q(p^-,{\mathbf w}) + \frac{p - p^-}{p^+ - p^-}\left(Q(p^+,{\mathbf w}) - Q(p^-,{\mathbf w})\right).
\end{gather}
In effect, $Q(p,{\mathbf w})$ is  the empirical $p$th quantile of the squared magnitudes of \({\mathbf w}\).

We define a neighbourhood as a set of indices corresponding to a subset of weights in the neural network layer. Let \({\mathcal N}_k\) represent the \(k\)th neighbourhood and let \(M\) represent the total number of neighbourhoods. The neighbourhoods in a layer form a partition of the set of weight indices, i.e., neighbourhoods are disjoint and the union of all neighbourhoods is the set of all weight indices. In practice, we define neighbourhoods so as to correspond to structures like kernels or filters in a neural network layer.
We also define a vector \(\bar{{\mathbf w}}\) containing the root mean squared weight value for each neighbourhood:
\begin{equation}
\bar{w}_k^2 = \frac{1}{|{\mathcal N}_k|}
\sum_{i\in {\mathcal N}_k}w_i^2.
\end{equation}

We use \(\ind[\cdot]\) to represent the indicator function, which takes the value 1 if its argument is true and 0 if it is false.

\subsection{Unstructured Pruning}
\label{sec:gp-unstruct}

Given the flexibility of Gibbs distributions, we have a massive range of Hamiltonians that could possibly work for pruning. In this section, we justify our choices of Hamiltonians for unstructured pruning by mathematically expressing requirements for pruning based on weight magnitudes, listing some heuristic design principles to help narrow down our search, and exploring some families of functions that can satisfy our requirements.

We use the heuristic that weights with lower magnitude are less important to the network's behaviour. So for unstructured pruning with a given \(p\), we design our Hamiltonian to converge to a state where the fraction \(p\) of weights with lowest magnitude are pruned. Let \({\mathbf x}_{\text{cvg}}\) represent this state:
\begin{equation}
	{\mathbf x}_{\text{cvg}} = \argmin_{\mathbf x} \left[{\mathbf x}^T{\mathbf w}^2\right] \text{ s.t. } \frac{\sum_i \ind[x_i = -1]}{N} = p,
\end{equation}
where \({\mathbf w}^2\) represents the element-wise square of \({\mathbf w}\). If there are multiple states satisfying these conditions, we select one arbitrarily.
To converge to this state, the Hamiltonian must therefore fulfil the requirement:
\begin{equation}
	\argmin_{\mathbf x} H({\mathbf x}) = {\mathbf x}_{\text{cvg}}.
\label{eq:unstruct_req}
\end{equation}
\({\mathbf x}_{\text{cvg}}\) can alternatively be expressed in terms of \(Q(p,{\mathbf w})\):
\begin{equation}
	x_{{\text{cvg}},i} = -1 + 2\ind[w_i^2 > Q(p,{\mathbf w})].
\end{equation}

We have a wide design latitude outside of this requirement. Design heuristics that guide our choice of Hamiltonians include:
\begin{itemize}
	\item Mathematical simplicity.
	\item Ease and speed of sampling from the resulting Gibbs distribution.
	\item Smooth convergence from random behaviour to the converged state. We hypothesize that this results in a better adapted network than sudden changes.
\end{itemize}
We now examine types of Hamiltonians that might make sense given these goals.

\subsubsection{Binary Hamiltonian}

An obvious simple Hamiltonian that fulfils (\ref{eq:unstruct_req}) is one that has a constant low energy when the mask is in the converged state, and a constant high energy otherwise:
\begin{equation}
	H({\mathbf x}) =  \alpha\ind[{\mathbf x} \neq {\mathbf x}_{\text{cvg}}] + \gamma, \quad \alpha > 0.
	\label{eq:bin_general}
\end{equation}
As previously discussed, constant additive factors in the Hamiltonian do not change the distribution, so \(\gamma\) does not matter. We can also combine \(\alpha\) into \(\beta\) when selecting our annealing schedule. Therefore, without loss of generality, we can consider:
\begin{equation}
	H({\mathbf x}) =  \ind[{\mathbf x} \neq {\mathbf x}_{\text{cvg}}].
	\label{eq:bin}
\end{equation}

Sampling from this distribution is easy, since it can be done by selecting \({\mathbf x}_{\text{cvg}}\) with some probability \(p_{\text{cvg}}\), and sampling uniformly from \(\{-1,1\}^N\) with probability \(1-p_{\text{cvg}}\). The exact probability is as follows.
\begin{lemma}
	Using a sampler for (\ref{eq:bin}) that selects \({\mathbf x}_{\text{cvg}}\) with probability \(p_{\text{cvg}}\) and  otherwise samples uniformly from \(\{-1,1\}^N\), \(p_{\text{cvg}} = \frac{1-e^{-\beta}}{(2^N-1)e^{-\beta}+1}\).
	\label{lemma:bin-p_cvg}
\end{lemma}

\begin{proof}
	The partition function normalizes the distribution, so it is:
	\begin{align}
		Z(\beta) &= \sum_{{\mathbf x}\in\{-1,1\}^N}e^{-\beta \ind[{\mathbf x}\neq{\mathbf x}_{\text{cvg}}]} \nonumber \\
				 &= (2^N-1)e^{-\beta} + 1.
	\end{align}
	The probability of sampling \({\mathbf x}_{\text{cvg}}\) is therefore:
	\begin{equation}
		P[{\mathbf x}={\mathbf x}_{\text{cvg}}] = \frac{1}{(2^N-1)e^{-\beta}+1}.
	\end{equation}
	Solving for \(p_{\text{cvg}}\) gives:
	\begin{align}
		p_{\text{cvg}} + \frac{1}{2^N}(1-p_{\text{cvg}}) &= P[{\mathbf x}={\mathbf x}_{\text{cvg}}] \\

		p_{\text{cvg}} &= \frac{1}{1-\frac{1}{2^N}}\left(\frac{1}{(2^N-1)e^{-\beta}+1}  - \frac{1}{2^N}\right) \nonumber \\
			   &= \frac{1-e^{-\beta}}{(2^N-1)e^{-\beta}+1}
	\end{align}
\end{proof}

\subsubsection{Linear Hamiltonians}

Another simple family of Hamiltonians that can meet our requirements are those that are linear in \({\mathbf x}\):
\begin{equation}
	H({\mathbf x}) = {\mathbf a}^T{\mathbf x}, {\mathbf a} \in \mathbb{R}^N.
	\label{eq:lin_family}
\end{equation}
Because they can take values other than zero and one, linear Hamiltonians offer more flexibility than a binary Hamiltonian and might provide smoother convergence.

A vector \({\mathbf x}\) minimizes the Hamiltonian if \(x_i=1\) when \(a_i<0\) and \(x_i=-1\) when \(a_i>0\). So, to fulfil (\ref{eq:unstruct_req}), \({\mathbf a}\) must be chosen such that \(a_i < 0\) when \(x_{{\text{cvg}},i} = 1\) and \(a_i > 0\) when \(x_{{\text{cvg}},i} = -1\). In terms of \(Q(p,{\mathbf w})\), the following must hold:
\begin{equation}
	\sgn(a_i) = \sgn(Q(p,{\mathbf w}) - w_i^2).
\end{equation}

Since only the sign of \({\mathbf a}\) is constrained and not its magnitude, we propose three possible schemes for selecting \({\mathbf a}\) based on our previously stated design considerations:
\begin{enumerate}[label=(\roman*)]
	\item \hfill%
		\(a_i = \sgn(Q(p,{\mathbf w}) - w_i^2),\)
			  \hfill\refstepcounter{equation}\textup{(\theequation)}%
		\label{eq:a1}
	\item \hfill%
		\(a_i = Q(p,{\mathbf w}) - w_i^2,\)
			  \hfill\refstepcounter{equation}\textup{(\theequation)}%
		\label{eq:a2}
	\item \hfill%
		\(a_i = \sqrt{Q(p,{\mathbf w})} - |w_i|.\)
			  \hfill\refstepcounter{equation}\textup{(\theequation)}%
		\label{eq:a3}
\end{enumerate}

In the formulation given by (\ref{eq:a1}) each vector element contributes either -1 or 1. Equations (\ref{eq:a2}) and (\ref{eq:a3}) instead cause \(a_i\) values to have higher or lower magnitudes depending on how far the corresponding weight magnitudes are from the \(p\)th quantile. We hypothesize that this will be more effective, since weights with very large or small magnitudes will cause the corresponding pruning mask elements to converge quickly to their final state, whereas more borderline weights will cause the corresponding pruning mask elements to behave more randomly, effectively holding off longer on the decision of whether to prune those weights until their importance becomes more clear. Equations (\ref{eq:a2}) and (\ref{eq:a3}) only differ in whether squared weight values or absolute weight values are used.

\subsubsection{Sampling}

Although Gibbs distributions are difficult to sample from generally, using a linear Hamiltonian causes the elements of \({\mathbf x}\) to be independent, meaning they can easily be sampled individually. In the following proposition, we show that our proposed Hamiltonian for unstructured pruning factors in terms of elements of \(x\).
\begin{proposition}
The Gibbs distribution corresponding to a linear Hamiltonian has a product form.
\label{prop:unstruct-factor}
\end{proposition}
\begin{proof}
	The partition function \(Z(\beta)\) normalizes the distribution. With a linear Hamiltonian (\ref{eq:lin_family}), it has the form:
\begin{align*}
    Z(\beta) &= \sum_{{\mathbf x}\in\{-1,1\}^N} e^{-\beta \sum_{i=1}^N a_ix_i} \\
    &= \sum_{x_1\in\{-1,1\}} \sum_{x_2\in\{-1,1\}} ... \sum_{x_N\in\{-1,1\}} \prod_{i=1}^N e^{-\beta a_ix_i} \\
    &= \prod_{i=1}^N \left(e^{-\beta a_i} + e^{\beta a_i} \right) \\
    &= \prod_{i=1}^N 2 \mbox{cosh} \Big(\beta a_i\Big).
\end{align*}
The Gibbs distribution therefore factors as:
\begin{align*}
p({\mathbf x}) &= \frac{1}{\prod_{i=1}^N 2 \mbox{cosh} \Big(\beta a_i\Big)} e^{-\beta \sum_{i=1}^N a_ix_i} \\
    &= \prod_{i=1}^N \frac{1}{2 \mbox{cosh} \Big(\beta a_i\Big)} e^{-\beta a_ix_i}.
\end{align*}
\end{proof}

\subsection{Structured Pruning}
\label{sec:gp-struct}

For structured pruning, we specifically consider the problem of pruning structured groups, or neighbourhoods, of weights which must be either all pruned or all kept. For instance, to do filter-wise pruning on a convolutional network, we would define the neighbourhoods to correspond to the filters of the layer. Following our approach of pruning based on weight magnitude, we propose pruning the neighbourhoods with lowest total squared weight magnitude, rather than considering individual weights. This mean that \({\mathbf x}_{\text{cvg}}\) is instead defined as:
\begin{multline}
	{\mathbf x}_{\text{cvg}} = \argmin_{\mathbf x} \left[{\mathbf x}^T{\mathbf w}^2\right] \text{ s.t. } \frac{\sum_i \ind[x_i = -1]}{N} = p \text{ and } \\ (i,j \in {\mathcal N}_k) \implies x_{{\text{cvg}},i} = x_{{\text{cvg}},j}.
\end{multline}
An equivalent definition in terms of \(Q(p,\bar{{\mathbf w}})\) is:
\begin{equation}
	i \in {\mathcal N}_k \implies x_{{\text{cvg}},i} = -1 + 2\ind[\bar{w}_k^2 > Q(p,\bar{{\mathbf w}})].
\end{equation}

A binary Hamiltonian can be used exactly as described above, just with this new \({\mathbf x}_{\text{cvg}}\). A linear Hamiltonian can also be used with \(a_i\) values that are 1 or -1 depending on the converged state of each \(x_i\):
\begin{equation}
	H({\mathbf x}) = \sum_{k=1}^M\sgn(Q(p,\bar{{\mathbf w}})-\bar{w}_k^2)\sum_{i\in{\mathcal N}_k}x_i.
	\label{eq:struct_ham_lin}
\end{equation}
This is analogous to the unstructured Hamiltonian (\ref{eq:a1}). The same approaches as previously discussed can be used for sampling these Hamiltonians for structured pruning.

\subsubsection{Quadratic Hamiltonian}

In structured pruning, the converged state is a function of average or total weight magnitudes over neighbourhoods. This means that setting linear coefficients based on the final converged state leads to coefficients that do not consider individual weight magnitudes, as the Hamiltonians (\ref{eq:a2}) and (\ref{eq:a3}) for unstructured pruning do. To both express structure requirements and consider the magnitudes of individual weights, we move to a quadratic family:
\begin{equation}
	H({\mathbf x}) = {\mathbf x}^T{\mathbf A}{\mathbf x} + {\mathbf b}^T{\mathbf x}, \, {\mathbf A} \in \mathbb{R}^{N\times N}, \, {\mathbf b} \in \mathbb{R}^N.
\end{equation}
This Hamiltonian is equivalent to the Ising model in statistical physics~\cite{cipra1987introduction}. Without loss of generality, we assume \({\mathbf A}\) is symmetrical. Since \(x_i \in \{-1,1\}\), the diagonal elements of \({\mathbf A}\) only contribute constant terms, so we ignore them. The off-diagonal elements of \({\mathbf A}\) add quadratic terms of the form \(a_{ij}x_ix_j\), which contribute an energy of \(a_{ij}\) if \(x_i\) and \(x_j\) take the same value and \(-a_{ij}\) if they are different. Therefore, the off-diagonal elements encourage two elements of \({\mathbf x}\) to either take the same value in sampling or to take different values. This allows the quadratic terms to express desired structures. Since they do not distinguish between the \(+1\) and \(-1\) states, only depending on whether two elements have the same state, the linear terms must be used for promoting or discouraging the pruning of particular elements.

Our desired structure requires mask elements in the same neighbourhood to take the same value. There are no cases where we require particular mask elements to take different values. Therefore, we set \({\mathbf A}\) such that \(a_{ij}<0\) if \(i\) and \(j\) are in the same neighbourhood, and \(a_{ij}=0\) otherwise.
For simplicity, we use a constant \(a_{ij}=-c/2\) for non-zero elements of \(A\). This leads to the Hamiltonian:
\begin{equation}
	H({\mathbf x}) = -c\sum_{k=1}^M\,\sum_{\substack{i,j \in {\mathcal N}_k\\i \neq j}}x_ix_j  + {\mathbf b}^T{\mathbf x}.
	\label{eq:struct_ham_b}
\end{equation}

To prove that the converged state obeys the requirements of structured pruning, we introduce the following property:

\begin{proposition}
	Given (\ref{eq:struct_ham_b}), for any \({\mathbf b} \in \mathbb{R}^N\), we can select a value \(c\) such that, in the state minimizing \(H({\mathbf x})\), all elements in the same neighbourhood take the same value.
	\label{prop:big-enough-c}
\end{proposition}

\begin{proof}
	Let \(N_\text{pairs}\) represent the number of pairs of weights in the same neighbourhood:
	\begin{equation}
		N_\text{pairs} = |\{\{i,j\} \mid \exists k, i,j \in {\mathcal N}_k\}|.
	\end{equation}
	Select a positive value \(c > (\max_{\mathbf x} {\mathbf b}^T{\mathbf x}) - (\min_{\mathbf x} {\mathbf b}^T{\mathbf x})\). A state where all elements in each neighbourhood take the same value therefore has maximum energy:
	\begin{equation}
		H({\mathbf x}) \leq -cN_\text{pairs} + \max_{\mathbf x} {\mathbf b}^T{\mathbf x}.
		\label{eq:min_c_proof}
	\end{equation}
	If there exist two elements in a neighbourhood with different values, then at least one \(-cx_ix_j\) term must switch from adding \(-c\) to the energy to adding \(+c\). Therefore, a state where some elements in a neighbourhood take different values has minimum energy:
	\begin{align}
		H({\mathbf x}) &\geq -c(N_\text{pairs}-2) + \min_{\mathbf x} {\mathbf b}^T{\mathbf x} \nonumber \\
			  &\geq -cN_\text{pairs} +2((\max_{\mathbf x} {\mathbf b}^T{\mathbf x}) - (\min_{\mathbf x} {\mathbf b}^T{\mathbf x})) + \min_{\mathbf x} {\mathbf b}^T{\mathbf x} \nonumber \\
			  &\geq -cN_\text{pairs} + (\max_{\mathbf x} {\mathbf b}^T{\mathbf x}) + c.
	\end{align}
	This is strictly greater than (\ref{eq:min_c_proof}) and therefore cannot minimize \(H({\mathbf x})\).
\end{proof}

This proposition shows that when selecting \({\mathbf b}\), we can tune \(c\) in a way that lets us limit our consideration to states where neighbourhoods take the same values. Within a neighbourhood \({\mathcal N}_k\), the linear contribution to the Hamiltonian is therefore \(\pm\bar{b}_k\) where \(\bar{b}_k = \sum_{i\in{\mathcal N}_k} b_i\). If this value is positive, then the neighbourhood will be pruned in the minimum state and if it is negative, the neighbourhood will not be pruned. Our definition of \({\mathbf x}_{\text{cvg}}\) therefore requires:
\begin{align}
\begin{split}
	\bar{b}_k < 0 &\text{ if } \bar{w}_k^2 > Q(p,\bar{{\mathbf w}}), \\
	\bar{b}_k > 0 &\text{ if } \bar{w}_k^2 < Q(p,\bar{{\mathbf w}}).
	\label{eq:bar_b_reqs}
\end{split}
\end{align}

To satisfy this requirement and incorporate individual weight magnitudes, we set \({\mathbf b}\) in a way analogous to (\ref{eq:a2}):
\begin{equation}
	b_i =  Q(p,\bar{{\mathbf w}}) - w_i^2.
\end{equation}
The corresponding Hamiltonian is then:
\begin{equation}
	H({\mathbf x}) = -c\sum_{k=1}^M\,\sum_{\substack{i,j \in {\mathcal N}_k\\i \neq j}}x_ix_j  + \sum_{i=1}^N\left(Q(p,\bar{{\mathbf w}}) - w_i^2\right)x_i.
	\label{eq:struct_ham}
\end{equation}
This formulation may perform better than the linear structured pruning Hamiltonian (\ref{eq:struct_ham_lin}) since it takes the magnitudes of individual weights into account before converging to a structured mask. We test both and compare results in the following section.

The hyperparameter \(c\) must be tuned for the particular network and task. Setting it too low will prevent the mask from converging to a state where neighbourhoods take the same value, meaning that structured pruning will not be achieved. On the other hand, setting \(c\) very high will result in the neighbourhood constraint always being followed, even very early in training, which will make the convergence to the pruning mask less gradual. We find that this reduces the final accuracy in practice.

\subsubsection{Sampling}
\label{sec:struct-sampling}

Sampling from the Gibbs distribution with a quadratic Hamiltonian is more challenging than for binary or linear Hamiltonians. First of all, we note that the distribution factors in terms of neighbourhoods and so individual neighbourhoods can be sampled independently:

\begin{proposition}
The Gibbs distribution corresponding to the Hamiltonian (\ref{eq:struct_ham}) has a block product form.
\end{proposition}
\begin{proof}
Since the first term of the Hamiltonian (\ref{eq:struct_ham}) is zero for elements from different neighbourhoods, the Hamiltonian can be expressed as:
\begin{align}
    H({\mathbf x}) &= \sum_{k=1}^M H_k({\mathbf x}_k),
\end{align}
where \(H_k({\mathbf x}_k)\) is the Hamiltonian (\ref{eq:struct_ham}) only computed over the elements of the neighbourhood \({\mathcal N}_k\). Following the same approach as for Proposition~\ref{prop:unstruct-factor}, the partition function can then be shown to factor as:
\begin{align}
    Z(\beta) &= \prod_{k=1}^M\,\sum_{{\mathbf x}_k \in \{-1,1\}^{|{\mathcal N}_k|}} e^{-\beta H_k({\mathbf x}_k)} \nonumber \\
    &= \prod_{k=1}^M Z_k(\beta),
\end{align}
and the overall distribution factors as:
\begin{align}
p({\mathbf x}) &= \prod_{k=1}^M\frac{1}{Z_k(\beta)} e^{-\beta H_k({\mathbf x}_k)}.
\end{align}
\end{proof}

If neighbourhoods are small enough, the partition function for each can be computed and the distribution can be sampled from directly. But for large neighbourhoods, this is computationally infeasible, and so we use a MCMC method to generate samples. A sampling method that parallelizes well and is simple to implement on a GPU is preferable for neural network training to highly iterative methods like standard Gibbs sampling or the Wolff algorithm~\cite{wolff1989collective}. We therefore propose using Chromatic Gibbs sampling~\cite{gonzalez2011parallel}.

Chromatic Gibbs sampling involves making a colouring over the elements in \({\mathbf x}\) such that any interacting elements \(x_i\) and \(x_j\) in the Hamiltonian have different colours. For our Hamiltonians, \(x_i\) and \(x_j\) interact if there is a term \(a_{ij} x_ix_j\) with $a_{ij} \neq 0$. An iteration of the Markov chain then consists of sampling all elements of one colour simultaneously given the current values of all other elements.

To create a colouring, we arbitrarily divide each neighbourhood into two sets of elements and modify the Hamiltonian to remove all quadratic terms containing elements in the same set. In graphical terms, this transforms the Markov random field for each neighbourhood from a complete graph to a complete bipartite graph that can be coloured with two colours, as shown in Figure~\ref{fig:chromatic}. This modification still preserves a high degree of connectivity within each neighbourhood and so is still effective at encouraging elements in neighbourhoods to take the same value. For instance, to perform filter-wise pruning of a convolutional network, we can divide all connections into two groups based on their input channel.

\begin{figure}[t!]
	\centering
	\begin{adjustbox}{width=0.35\textwidth}
        \input{bipartite.tex}
	\end{adjustbox}
	\captionsetup{font=small}
	\caption[Removing interactions to allow a 2-colouring of a random field.]{Removing interactions between elements so that neighbourhoods form a complete bipartite graph allows a 2-colouring, which in turn allows chromatic Gibbs sampling to be used.}
	\label{fig:chromatic}
\end{figure}

When using MCMC sampling, it is important to initialize the chain at a plausible configuration, since later samples may still be somewhat correlated with the initial state~\cite{brooks2011handbook}. To initialize the chain, we sample from an approximation of the quadratic Hamiltonian that ignores states that violate the structure requirement. This makes the quadratic term a constant value, leading to the linear Hamiltonian:
\begin{equation}
	H(\bar{{\mathbf x}}) = C + \sum_{k=1}^M |{\mathcal N}_k|Q(p,\bar{{\mathbf w}}) - \bar{{\mathbf w}}_k^2)\bar{x}_k,
	\label{eq:struct_ham_approx}
\end{equation}
where \(\bar{x}_k\) is the value taken by all \(x_i\) in neighbourhood \(k\) and \(C\) is a constant representing the contribution of the quadratic terms. Since it is linear, this Hamiltonian is easy to sample from, and is likely to represent local modes of the distribution since the quadratic term penalizes states that do not obey the structure requirement.

\subsection{Schedule Stretching}
\label{sec:gp-schedule}

Many existing pruning methods make use of much longer training schedules than are typically used for training networks because they do additional fine-tuning (e.g., \cite{guo2016dynamic,han2016deep,li2017pruning}). Our pruning method aims to prune and train at the same time so we avoid explicit fine-tuning, but we still investigate whether a longer training schedule can provide benefits. To test this, we propose \textbf{schedule stretching}. During training, we follow schedules for setting the optimizer learning rate and the Gibbs distribution parameter \(\beta\) as a function of the current training epoch. Let \(n\) represent the current epoch number, starting from zero, and let \(\lambda(n)\) and \(\beta(n)\) represent the learning rate and \(\beta\) schedules, respectively. When using schedule stretching, we set a hyperparameter \(s\) to a positive integer value, and instead use \(\lambda(\lfloor \frac{n}{s} \rfloor)\) and \(\beta(\lfloor \frac{n}{s}\rfloor)\) to set the learning rate and \(\beta\) at each epoch. The total number of training epochs is increased by a factor of \(s\).

\section{Introduction}
\label{sec:introduction}

Over the past decade, deep learning models have shown remarkable success in many types of data analysis, including image, speech, and text processing. But while they are powerful, state-of-the-art deep neural networks have grown massively in complexity. Using current models on resource-limited devices like smartphones, routers, other edge devices, and embedded devices is often impossible. Even running them at scale on more powerful computers can be prohibitively costly.

Neural network pruning is one way to alleviate this problem. Pruning techniques remove connections or weights within neural networks in such a way that inference accuracy is minimally affected. In many deep neural networks, the majority of connections can be removed without significantly reducing accuracy, which can speed up inference and reduce memory requirements considerably.

Neural network pruning methods can be divided into structured and unstructured pruning. In unstructured pruning, any set of connections can be removed. Structured pruning instead requires that the pruned connections follow some structure, such as removing entire rows from weight matrices or entire filters from convolutional layers. Structured pruning can usually improve performance metrics like inference speed at lower sparsity levels than unstructured pruning, since the structure can be taken advantage of in optimizing performance. On the other hand, unstructured pruning can generally remove more connections without significantly decreasing accuracy.

This paper introduces Gibbs pruning, a family of neural network pruning methods for structured and unstructured pruning that take inspiration from statistical physics.
We apply the concept of Gibbs distributions, which describe stochastic models that take the form of random fields. Gibbs distributions are highly flexible in terms of network properties that they can express, and quadratic energy functions, such as Ising models, can capture parameter interactions to induce structures required for structured pruning. Past work in machine learning and signal processing has applied Gibbs distribtions or similar concepts for parameterizing models~\cite{ackley1985learning}, image restoration~\cite{geman1984stochastic}, and optimization~\cite{kirkpatrick1983optimization}. To the best of our knowledge, Gibbs pruning is the first application of Gibbs distributions for neural network pruning.

Gibbs pruning uses an approach related to stochastic regularization methods like dropout~\cite{hinton2012improving,labach2019survey} in adding random behaviour during training to push the network towards a desirable representation. We induce a Gibbs distribution over the weights of a neural network and sample from it during training to determine pruning masks. This procedure leads to a learned network structure that is resilient to high degrees of pruning. We also take advantage of the temperature parameter in the Gibbs distribution to anneal the distribution, gradually converging to a final pruning mask during training that is well-adapted to the network.

The original contributions to neural network pruning research in this paper are as follows:

\begin{itemize}
	\item A versatile framework for expressing, developing, and understanding neural network pruning methods that connects them to stochastic regularization methods and concepts from statistical physics.
	\item A set of novel neural network pruning methods for both unstructured and structured pruning that outperform existing methods. We establish a new state-of-the-art result for pruning ResNet-56 with the CIFAR-10 dataset.
\end{itemize}

An earlier version of this work has been released as a preprint~\cite{labach2020framework} and has been accepted for publication in the 2020 IEEE Global Communications Conference (GLOBECOM). Changes from the earlier work include proposing more possible Hamiltonian functions, discussing their mathematical properties in more detail, and comparing their performance, as well as providing an extended literature review and more detailed analysis of our results.

This paper is organized as follows. Section \ref{chap:related} is a survey of relevant related works in the area of neural network pruning. Section \ref{chap:theory} gives a formal description of the Gibbs pruning framework and explores various possible pruning methods that can defined within this framework from a mathematical perspective. Section \ref{chap:evaluation} presents the results of various experiments carried out to explore the design space of Gibbs pruning methods and to compare them to existing pruning methods. Finally, Section \ref{chap:conclusion} summarizes our main findings and suggests possible future research directions.

\section{Related Work}
\label{chap:related}

A wide variety of methods for accelerating neural networks by modifying models have been proposed. Broadly speaking, they fall into the following categories.

\begin{itemize}
	\item \textbf{Network pruning}: The removal of connections during or after training.
	\item \textbf{Small models}: Neural network architectures that have been manually designed to approach the performance of large networks while having many fewer parameters, e.g., \cite{howard2017mobilenets, iandola2016squeezenet}.
	\item \textbf{Weight quantization}: Limiting the set of values that weights can take to reduce their memory usage and/or speed up mathematical operations, e.g., \cite{hubara2017quantized,han2016deep}.
	\item \textbf{Tensor decomposition}: Approximating linear transformations within neural networks as products of smaller tensors, e.g., \cite{denton2014exploiting,denil2013predicting}.
	\item \textbf{Knowledge distillation}: Training a small neural network to approximate the function implemented by a large, trained neural network, e.g., \cite{hinton2015distilling,romero2014fitnets}.
\end{itemize}

In practice, multiple methods can be used together to further improve performance. For instance, in \cite{han2016deep}, the authors propose using network pruning alongside weight quantization and Huffman coding to massively reduce the storage requirements of a neural network.

Neural network pruning research dates back to at least the 1980s~\cite{sietsma1988neural}. Since 2015, there has been a resurgence of neural network pruning research focused on developing methods for modern DNNs, which is the line of research that this paper falls into. In this section, we classify existing pruning methods in a number of ways to explain how our proposed method relates to contemporary research.

\subsection{Structured vs. Unstructured Pruning}

Unstructured pruning methods \cite{frankle2019stabilizing,frankle2019lottery,zhu2017prune,guo2016dynamic,han2015learning,lecun1990optimal,dong2017learning,molchanov2017variational,louizos2018learning} are generally usable for any neural network topology, since they are not limited in terms of which connections they prune. Structured pruning methods \cite{li2017pruning,luo2017thinet,molchanov2016pruning,he2018soft,he2019filter,he2017channel,hu2016network,wen2016learning,liu2017learning,ye2018rethinking,yu2018nisp,zhuang2018discrimination,hacene2019attention} are more specific to particular topologies, with nearly all recent methods being designed for CNNs. The most common structures to prune are entire convolutional filters or channels, effectively reducing the size of one tensor dimension for both weight and activation tensors as previously discussed. However, other structures are possible, including pruning entire convolutional kernels (e.g., \cite{molchanov2016pruning}) or pruning all but one connection in a kernel, allowing the convolution operation to be replaced by an operation that simply scales and shifts the input~\cite{hacene2019attention}.

While some existing pruning methods can be used for both structured and unstructured pruning \cite{gomez2019learning,he2018amc}, the vast majority are specialized for one or the other task. Our proposed method defines a framework that can be used for unstructured or structured pruning with arbitrary structures within a layer, making it much more versatile than most existing pruning methods.

\subsection{Post-training vs. In-Training Pruning}

Most pruning methods take a trained neural network as an input, prune some connections, and then \textbf{fine-tune} the network: repeating the training procedure for a number of epochs to recover the lost performance \cite{frankle2019lottery,frankle2019stabilizing,guo2016dynamic,han2015learning,lecun1990optimal,li2017pruning,luo2017thinet,molchanov2016pruning,dong2017learning,he2018amc,he2017channel,hu2016network,yu2018nisp,zhuang2018discrimination}. Pruning and fine-tuning may be repeated multiple times. In contrast, some pruning methods start with an untrained network and perform pruning or adapt the network for pruning during the training procedure \cite{zhu2017prune,gomez2019learning,wen2016learning,he2018soft,he2019filter,liu2017learning,ye2018rethinking,hacene2019attention,molchanov2017variational,louizos2018learning}. We call these approaches \textbf{post-training} and \textbf{in-training} pruning respectively. When a trained network is available, post-training pruning methods usually have an advantage in that fine-tuning is faster than training from scratch would be. But when a trained network is not available, in-training pruning methods will usually be faster since they do not add additional training epochs for fine-tuning. Our proposed method uses in-training pruning, since we believe that this approach can help the training procedure arrive at a final representation that is more robust to pruning than an ordinary trained network would be.

\subsection{Pruning Heuristics}

A central part of neural network pruning research has been developing metrics for deciding how important a connection is to a network's performance. The simplest and most common heuristic is weight magnitude: connections with weights close to zero are pruned \cite{frankle2019lottery,frankle2019stabilizing,zhu2017prune,gomez2019learning,guo2016dynamic,han2015learning,li2017pruning,wen2016learning,he2018soft,he2018amc}. More advanced heuristics have also been developed, such as using the Hessian of the loss with respect to the weights \cite{lecun1990optimal,dong2017learning}, a Taylor expansion approximation of the loss with respect to the weights~\cite{molchanov2016pruning}, and measuring the redundancy of parts of the network~\cite{he2019filter}. Other pruning methods have used sample inputs from the training set to estimate the importance of parameters~\cite{luo2017thinet,hu2016network}.

In \cite{gale2019state}, the authors compare methods using more advanced heuristics to the approach of pruning based on weight magnitude and find that the more complex heuristics do not provide an advantage in practice. This is in line with a number of recent papers that have gone back to using weight magnitude and have focused instead on changing how and when pruning is applied to the network~\cite{gomez2019learning,frankle2019lottery,frankle2019stabilizing}, which has arguably proved to be a more important concern. Our proposed method follows this approach.

~

On the whole, the most closely related method to Gibbs pruning is targeted dropout~\cite{gomez2019learning}, which is also an in-training pruning method based on weight magnitude that can be used for structured or unstructured pruning. Like our proposed method, it takes inspiration from stochastic regularization methods like dropout in stochastically pruning during training to make the network robust under pruning. We compare our method to it experimentally in Section~\ref{chap:evaluation}.

\subsection{Retraining with Random Initialization}

In \cite{liu2019rethinking}, the authors made the surprising observation that when using many established pruning methods, if the pruned network weights were re-initialized randomly and trained from scratch, the resulting accuracy would match or surpass that of the pruned network. While existing pruning methods were effective at determining effective small architectures, it turned out to be more effective to train these architectures normally rather than starting with a large network and pruning it. This finding was a major challenge to the value of network pruning as a compression technique.

A number of recent papers have replicated this experiment with new pruning methods and have shown that they outperform retraining with random initialization, in particular methods like ours that focus more on co-adapting network weights and pruning masks during the training and pruning procedures~\cite{gomez2019learning,frankle2019stabilizing,gale2019state}. We also demonstrate this result with our proposed method in Section \ref{sec:ev-re-initialization}, making it part of a new generation of pruning methods that can be shown to outperform random re-initialization.

redpajama_set_name: RedPajamaArXiv


We offer a range of add-on services to complement the website you have with us.
Want to get placed higher on search engines like Google and Bing but don't know where to start? Not to worry, our in-house experts will tweak every bit of your website to make sure your business shows up where it belongs: on the top!
We'll help you with modernizing and re-writing your content so you can capture those sweet leads! Not only will it help with your business' recognition, but you will also see an uptick in your prospective clients!
Do you feel like it's about time you got a new logo to keep up with the ever-changing preferences of your audience? If your answer is yes, our talented designers can help you reflect and communicate your core message to your audience.
We can help take your business to the next level and keep your marketing costs low by being your trusted marketing department. We offer an All-in-One Marketing Consultation Package as well as our Social Media Supercharge Package.


Dot pendant from woud. Made of perforated metal and opal glass. The small holes allow light to penetrate and create a cozy and playful atmosphere. Dot is available in white and black and in three different sizes. They can advantageously be mixed and matched to create lovely bright spots in the home.


text:
Purdue Women's Basketball
Purdue Women's Basketball Announces 2020-21 Big Ten Schedule
Both Men's and Women's College Basketball is here soon!
By dubjell6 Nov 23, 2020, 5:22pm EST
Share All sharing options for: Purdue Women's Basketball Announces 2020-21 Big Ten Schedule
Photo by G Fiume/Maryland Terrapins/Getty Images
Sharon Versyp and the Purdue Women's Basketball program have now finalized their 2020-21 schedule as the Big Ten announced the conference slate for the upcoming season!
The Boilermakers will play just 5 non-conference games ahead of the Big Ten season against North Alabama (Nov 29th), Miami (Ohio) (Dec 3rd), Valparaiso (Dec 6th), Buffalo (Dec 9th), and Bowling Green (Dec 13th) all inside the confines of Mackey Arena.
They will play a home and home against Indiana, Rutgers, Maryland, Illinois, Penn State, Michigan and Michigan State.
The home game lineup includes dates with Northwestern, Nebraska, and Wisconsin, along with the aforementioned Indiana, Rutgers, Maryland, Illinois, Penn State, Michigan and Michigan State. Purdue will also travel to Minnesota and Ohio State.
Purdue will have a bit of a lighter schedule at the beginning of conference play with Penn State, Nebraska and Michigan State all coming off down years before ramping it up against the likes of Rutgers, Maryland, Indiana and Iowa.
We will learn a lot about this team in the first couple weeks of the season as the Big 3 of McLaughlin, Harris and Oden are all gone from last year's squad.
It's happening!
Check out our 2⃣0⃣-game #B1G slate. pic.twitter.com/95TmFQL4ur
— Purdue Women's Bball (@PurdueWBB) November 23, 2020

redpajama_set_name: RedPajamaCommonCrawl


text:
China jails 'Frankenstein' scientist who shocked the world by creating the first ever gene-edited babies
Patrick Knox
Updated: 30 Dec 2019, 15:53
A ROGUE Chinese scientist who made the world's first genetically edited babies was today sentenced to three years in prison because of his illegal research.
He Jiankui's experiment – branded "monstrous" by some researchers – led to the birth of gene-edited twin girls nicknamed Lulu and Nana.
He Jiankui's claims that he created the world's first gene-edited babies are trueCredit: AP:Associated Press
The birth of the world's first gene-edited babies was announced in November
The babies' birth stunned the world and sparked a global debate over the ethics of gene editing.
Under Chinese law scientists are banned from DNA tinkering.
Jiankui was fired from his research job at China's Southern University of Science and Technology as a result of the finding.
But things became worse for Jiankui when he was arrested and convicted of practising medicine without a license.
Today he was locked up for three years as well as being fined £327,000 by a court in the southern city of Shenzhen, China's official Xinhua News Agency reported.
Two other researchers involved in the project received lesser sentences and fines.
The verdict said the three defendants had not obtained qualification as doctors, pursued fame and profits, and deliberately violated Chinese regulations on scientific research.
This crossed an ethical line in both scientific research and medicine, according to Xinhua.
What is gene editing?
Gene editing involves inserting, deleting, modifying or replacing the DNA of a living organism
It could help protect humans against diseases and natural illness
And some believe that gene editing could give us superhuman enhancements
These could include boosted sense of smell or improved night vision
Scientists are divided over the ethics of gene editing
Editing DNA is a new and risky science that could have unknown consequences for babies
And DNA-editing could lead to societal changes, with the prospect of "designer babies" becoming real
This could lead to severe inequality – a future where only the rich can afford to modify their DNA could lead to a new elite race of superhumans
Gene editing has been successfully performed on mice embryos
And one Chinese scientists claims to have successfully gene-edited the embryos of two human baby girls
The court said the researchers were involved in the births of three gene-edited babies to two women, confirming reports of a third baby.
It said all three pleaded guilty during the trial, which Xinhua reported was closed to the public because of privacy concerns.
Jiankui, the lead researcher, shocked the scientific world when he announced in November 2018 that he had altered the embryos of twin girls who had been born the same month.
He described his work in exclusive interviews with The Associated Press.
He said he had used a tool called CRISPR to try to disable a gene that allows the AIDS virus to enter a cell, in a bid to give the girls the ability to resist the infection..
The CRISPR tool has been tested elsewhere in adults to treat diseases, but many in the scientific community denounced his work as medically unnecessary and unethical, because any genetic changes could be passed down to future generations.
SCHUEY BLOW
Schumacher is 'very deteriorated and not how we remember him', surgeon claims
BIRTH TRAGEDY
Woman, 30, charged with MANSLAUGHTER after her baby died as she gave birth
HOLOCAUST HORROR
Full horror of Auschwitz laid bare in harrowing colourised pics
The US forbids editing embryos except for lab research.
Jiankui, who is known as JK, told AP in 2018 that he felt a strong responsibility to make an example, and that society would decide whether to allow the practice to go forward.
He disappeared from public view shortly after he announced his research at a conference in Hong Kong 13 months ago, apparently detained by authorities, initially in an apartment in Shenzhen, a city in Guangdong province that borders Hong Kong.
It wasn't clear if the three-year prison term includes any of the time he has already spent in Chinese custody.
A Chinese scientist said the sentence should have been harsher to deter others.
Kehkooi Kee, a Tsinghua University researcher who conducts gene-editing research on stem cells, also said that He should be held responsible for any fallout from the experiment on the lives of the babies and their families.
The controversial scientist said he altered the DNA of twin baby girlsCredit: AP:Associated Press
Jiankui and his team faced legal repercussions for their innovative workCredit: AP:Associated Press
This is a microplate containing embryos injected with the Cas9 protein, a DNA-editing proteinCredit: AP:Associated Press
A monitor showing a video feed of Chinese scientists editing an embryo using DNA proteinsCredit: AP:Associated Press
'Monstrous' monkey CLONES gene-edited to suffer diseases have been born in China


text:
Q: Python For OpenCV2.4 I downloaded opencv 2.4 source code from svn, and then I used the command 'cmake -D BUILD_TESTS=OFF' to generate makefile and make and install. I found that the python module was successfully made. But when I import cv2 in python, no module cv2 exists. Is there anything else I should configure? Thanks for your help.

A: You should either copy the cv2 library to a location in your PYTHONPATH or add your current directory to the PYTHONPATH.

A: Have you run 'make install' or 'sudo make install'? While not absolutely necessary, it copies the generated binaries to your system paths.

redpajama_set_name: RedPajamaStackExchange


text:
Criticism Quotes
Home » Criticism Quotes
"When art critics get together they talk about Form and Structure and Meaning. When artists get together they talk about where you can buy cheap turpentine."
"People tend to criticize their spouse most loudly in the area where they themselves have the deepest emotional need."
– Gary Chapman
"The pleasure of criticizing takes away from us the pleasure of being moved by some very fine things."
"You sounded like Dolly parton on helium."
(After kristy lee cook of season 7 on american idol,sang her country rendition of the Beatles'"Eight Days A Week.)"
– Simon Cowell
"You need a high degree of corruption or a very big heart to love absolutely everything"
– Gustave Flaubert, November
"Remember: when people tell you something's wrong or doesn't work for them, they are almost always right. When they tell you exactly what they think is wrong and how to fix it, they are almost always wrong."
– Neil Gaiman
"How much easier it is to be critical than to be correct."
– Benjamin Disraeli
"I've been all over the world and I've never seen a statue of a critic."
– Leonard Bernstein
"The criminal is the creative artist; the detective only the critic."
– G.K. Chesterton
"I criticize by creation, not by finding fault."
"Anyone who has the temerity to write about Jane Austen is aware of [two] facts: first, that of all great writers she is the most difficult to catch in the act of greatness; second, that there are twenty-five elderly gentlemen living in the neighbourhood of London who resent any slight upon her genius as if it were an insult to the chastity of their aunts."
– Virginia Woolf
"The aim of language…is to communicate…to impart to others the results one has obtained…As I talk, I reveal the situation…I reveal it to myself and to others in order to change it."
"Criticism is just someone else's opinion. Even people who are experts in their ?elds are sometimes wrong. It is up to you to choose whether to believe some of it, none of it, or all of it. What you think is what counts."
– Rodolfo Costa
"That was excellently observed', say I, when I read a passage in an author, where his opinion agrees with mine. When we differ, there I pronounce him to be mistaken."
– Jonathan Swift
"Everything in our background has prepared us to know and resist a prison when the gates begin to close around us . . . But what if there are no cries of anguish to be heard? Who is prepared to take arms against a sea of amusements? To whom do we complain, and when, and in what tone of voice, when serious discourse dissolves into giggles? What is the antidote to a culture's being drained by laughter?"
– Neil Postman
"Democracy is the worst form of government, except for all the others."
– Winston S. Churchill
"You can't let praise or criticism get to you. It's a weakness to get caught up in either one."
– John Wooden
"But instead of spending our lives running towards our dreams, we are often running away from a fear of failure or a fear of criticism."
– Eric Wright
"It's far easier to write why something is terrible than why it's good. If you're reviewing a film and you decide "This is a movie I don't like," basically you can take every element of the film and find the obvious flaw, or argue that it seems ridiculous, or like a parody of itself, or that it's not as good as something similar that was done in a previous film. What's hard to do is describe why you like something. Because ultimately, the reason things move people is very amorphous. You can be cerebral about things you hate, but most of the things you like tend to be very emotive."
– Chuck Klosterman
"Eccentricity is not, as some would believe, a form of madness. It is often a kind of innocent pride, and the man of genius and the aristocrat are frequently regarded as eccentrics because genius and aristocrat are entirely unafraid of and uninfluenced by the opinions and vagaries of the crowd."
– Edith Sitwell
12345Next »Last »
Being Yourself
Debasish Mridha
DesiQuotes.com
Slogans Motto
QuotesGuy


Stage 17 illustrates the Tour de France's willingness to innovate. The second of three days in the Pyrenees, it has the potential to go down as one of the most exciting stages of the 2018 edition—and, maybe, in race history. At only 65K, it's the Tour's shortest road stage since the late 1980s. But it covers three major climbs, including the highest summit this year, and has an unorthodox setup: Instead of starting the stage in one group, riders will line up on a starting grid according to their places in the General Classification.
The day begins on the base of the Category 1 Montée de Peyragudes, an ascent riders should know well after climbing most of it during last year's Tour. A short downhill then takes them to the base of the next climb: the shorter but steeper Category 1 Col de Val Louron-Azet. But the real fun comes on the climb to the finish: the Hors Categorie Col du Portet. Making its Tour debut, the Portet is the highest summit this year and is certain to decimate the field.
How will teams handle the unconventional start format and stage profile? They'll be climbing right from the gun, but given the defensive tactics we've seen in past Tours, don't be surprised if things begin slow. Riders and teams will likely regroup on the lower slopes of the first climb, waiting for a later opportunity to launch serious attacks.
It all comes down the Col du Portet, a climb similar in length and average gradient to Mont Ventoux. The final summit finish of 2018, it might be the last chance for the competition to try and crack Team Sky.
Stage 17 could provoke a timely GC battle between Tom Dumoulin and Team Sky.
This stage is too short and hard not to tempt one of the GC contenders. Overall leader Geraint Thomas has won two summit finishes so far and shows no signs of weakness. Chris Froome hasn't looked as strong as in previous years, but he's likely timed his peak for the final week.
What about Tom Dumoulin? The Dutchman is a master of riding his own race, setting a steady tempo, and returning to the front even after he's been gapped. He's the most dangerous rider in the eyes of Sky, and the team will certainly try to drop him.
When it comes to provocateurs, look for Primoz Roglic, Steven Kruijswijk, and Romain Bardet to animate the race early. Last but not least, there's Team Movistar, who despite coming to the Tour with three leaders has had little impact on the race overall. Mikel Landa and Nairo Quintana are its best-placed riders right now, and tomorrow is the team's last chance to contend for a podium finish.
The stage begins at 9:15 a.m. EST and is worth watching from start to finish. No matter what, make sure you're watching as the race hits the Col du Portet around 10:35 a.m.


text:
Survey, interviews serve as basis for in-depth report
By Dr. William D. Downs Jr.
That's how the entrance signs to the city have read for more than 30 years. Why doesn't Arkadelphia grow? That was the question I posed to students as a class project this spring for the Advanced Reporting class. And for the next three days you will read the results of an in-depth report that has succeeded in getting these six students out of the classroom and into a real-life situation. It has been my pleasure to serve as the students' managing editor.
The information for the three-part series was gathered in three ways: First, 150 open ended surveys were mailed to a cross-section of Arkadelphia residents, which generated 79 responses, a highly respectable 53 percent return. Second, townspeople were invited to visit with the class, which met twice a week. During these interviews, students gathered additional information. The third step was to visit as many as possible of the survey respondents who had indicated on their response that they would be willing to be
interviewed.
In addition, an e-mail survey of all OBU students generated 348 responses. The last two weeks of the semester have been devoted to organizing the responses in what we hope you will find to be a highly readable format. Rather than structure the responses in a standard news story, we decided it would be best to present the information in "blocks," most of which begin with a bold-faced subhead to give readers an idea about the content of each section.
Because of the volume of questionnaires that were completed and returned, it was not possible to include statements from all the respondents. But as each response was received and each interview was submitted, the major trends began to appear early in the process: Jobs were No. 1 but there was much more, with all factors ultimately coming together to provide a road map for growth. The liquor issue turned out to be no issue at all. It was obvious that respondents were clearly aware that there were many other more pressing issues that needed to be addressed.
Why did we do this? One reason was that it was a highly unusual civic journalism project that would push students out of their comfort zones to conduct interviews with Arkadelphia residents who share our concern for the future of our city. A second reason was that we really cared-professor and students alike. Which provides a third reason for the project: What the students did could actually make a difference. A fourth reason was one that we did not expect at the start of the semester: With the organization of a countywide Strategic Planning Commission, led by Dr. Wesley Kluck, the research would not only be published in the Daily Siftings Herald but would also be used as a foundation for the Commission's planning. There is a fifth reason, too. I've just finished my 40th year at OBU-that's 80 semesters. And Vera and I love it here. So do the students. But we must take action.
Admittedly, the students became increasingly apprehensive as the surveys were received and interviews were conducted. "Will this hurt anyone's feelings?" I remember one student asking. And the answer is "no." The purpose of the study and the stories that will run this week is to motivate all of us – Arkadelphia-area residents and students – to join in a united effort to prove that Arkadelphia CAN grow and that it can become an even better place to call home.
Brown Hardman, former chair of the CCIC, may have said it best: "If you always do what you've always done, you'll always be where you are because where you are right now is a result of what you have or have not done."
Dr. William D. Downs Jr. is chair of the Department of Mass Communication at Ouachita Baptist University.


text:
Archive:Flatland
From Oddworld Library
Host: Videogames.com
Author: Paul O'Connor
The Designer Diary
Part Four: Flatland
By Paul O'Connor
Reviewers generally gushed over the lavish art and clever gameplay of our first two games, but they qualified their praise with a dreaded classification: 2D. That's 'two-dee,' as in two-dimensional, as in platform game, as in side scroller, as in… you get the idea. We wore that badge proudly, as we felt the side-view format was dynamite for the kind of gameplay we wanted to do, and it also helped our inaugural products stand out from a hoard of me-too 3D titles. But even while we were polishing our engine in the closing days of Exoddus, we knew 3D was in our future.
It's a future that couldn't come soon enough for many Oddworld Inhabitants. While we've made our mark by breathing new life into the 2D format, the fact is that Oddworld has always been a 3D company. Our company was founded by Sherry McKenna and Lorne Lanning, pioneers in the 3D industry. Practically everything in our games was modeled in 3D. In an ideal world, we would have come out of the gate with a 3D game, but the hardware required to realize even a portion of what we wanted to do wasn't available when we started building Oddworld.
All that has changed with the release of the latest video-game platforms. As you probably know, our next title, Oddworld: Munch's Oddysee, will be a 3D game. What does that mean for the people building the game? Well, a lot and a little.
It means a little because we've already got a 'digital backlot' of 2D characters and places from our first two games, and transitioning them into a 3D game (while not without some teething troubles) has been pretty painless. It means a lot because the people that I work with—the game designers—have had to cut loose of their 2D mindset and look at the Oddworld play experience in a whole different light. We've bid farewell to Flatland, and the future is a labyrinth of extruded polygons and inverted normals.
Those are just a couple of terms we designers are wrestling with as we learn a new set of tools for assembling Munch's Oddysee. Fortunately, some of our guys have experience with 3D, and they're serving as pathfinders by setting up tutorials for us knuckle-dragging throwbacks (meaning me, mostly) who are new to this environment. We'll figure it out. This stuff isn't rocket science.
(Rocket science is easier).
What is proving tougher is getting our brains wrapped around the gameplay differences between 2D and 3D. Munch's Oddysee isn't a 2D game jumped up to 3D; it's a whole new experience. Many common mechanics from our 2D games—like pits or meat saws—take on an entirely different aspect in 3D, where you can just walk around most hazards. We've had to rethink much of our artificial intelligence, as the facing of enemies (and what they can 'see') is much less clear-cut. The list goes on and on….
So what are we strangers in this strange land doing to keep from wandering astray? We're throwing paper at it, of course… and I'll tell you all about it next month.
NEXT: Getting the Treatment
Retrieved from "https://oddworldlibrary.net/oddenc/index.php?title=Archive:Flatland&oldid=1869"
Content is available under Attribution-NonCommercial-NoDerivs 3.0 Unported unless otherwise noted.


Happy April to you all and here's hoping we start getting some Spring weather!!
At least our menu is seasonal and delicious. If you haven't yet been into the restaurant since the launch of the Spring menu last week, then don't miss out. We have all sorts of delicious dishes to tempt you into our Devon Restaurant. Pictured are some of our main courses Monkfish, Pork, Arancini and Lamb. Full descriptions are in the menu below. Book now for dinner Wednesdays to Saturdays and lunches Wednesdays to Sundays. Please note our lunch menu is not the same as the Dinner Menu below and Sundays is The Big Sunday Lunch!


It is a new concept of multi-risk protection fully developed by Techs – Textil Santanderina. Stands out, above all its qualities, the excellent performances against the risks of electric arc and flash fire..
Due to its newfangled composition, Techsalexandra offers excellent performances in use, maximum values of resistance, abrasion and pilling are guaranted, as well as high permanent fire protection.
Techsalexandra is available in a wide variety of fabrics, weights, colors and finishes. It is perfectly adapted to the different needs of the market and the end user. It has an extensive certification, and a large stock to provide immediate delivery service.


Some artists constantly change up their formula, while others are content. There is nothing wrong with being content, but sometimes it puts your artistry into serious questioning. There are many artists I could characterize as 'one dimensional', but I chose five who truly epitomize this description. And yes, some of these are artists I highly respect and listen to. I'm a meanie, I know.
I really do respect and enjoy India. Arie's music, but I feel that she hasn't truly 'spread' her wings for a couple of albums. I'm all about "if it ain't broke, don't fix it", but still… "Video" will always be a classic, but can the same be said of "Cocoa Butter" from 2013's quietly released SongVersation? Arie is one of those artists who's so 'chill' that she sometimes comes off a bit boring and blasé. She remains prudent mind you, but I wouldn't mind if she shook things up just a wee bit.
I don't know about everyone else, but frankly I'm pretty tired of hearing about Wiz smoking weed. Because of this lack of range (or artistry), Wiz's sophomore album O.N.I.F.C. was incredibly forgettable. As I've referenced earlier, his song "It's Nothin'" featuring 2 Chainz truly was 'nothing'. Of course, the lack of another banger like "Black & Yellow" would hurt anybody. Still, no pass Wiz!
Before I get stoned, this doesn't merely apply to One Direction. Why are they highlighted? Well they are easily the most popular boy-band / teen pop artist(s) not named Justin Bieber. Basically, the formula for the teen pop band is what makes the majority of them one dimensional. The format is so tried and true that in many respects, it hurts the artistry. We've come to expect extraordinarily good-looking girls/guys and corn-ball/schmaltzy songs slated somewhere between pop, R&B and rock. Sure, One Direction's Take Me Home was a good album contextually considering its fan base, but other than teen girls, who's really listing them as their favorite 'artist'? Exactly.
Last year I wouldn't have listed The Weeknd on this list, but Kiss Land had a hand in this change of heart. I enjoy listening to The Weeknd… on Trilogy. As for Kiss Land, he boxes himself into the same thing he did on his mixtapes only he does it much less satisfyingly. The voice is still incredible, but how many more times do you sing about weed consumption, being oversexed, so on and so forth? I'd say the same about The-Dream, whose IV Play was clumsy as albeit and lacked the fun of the artists previous efforts.
Isn't this one obvious? This man goes dumber than fudge… replace 'fudge' with another word. I'm not gonna lie, I own both of his albums and was even onboard during his "Duffle Bag Boy" days when he was Tity Boi in Playaz Circle, but that doesn't mean I don't consider him 'one dimensional'. That said, can anyone really imagine ole boy being truly serious? Nope, didn't think so.


In a Detroit suburb today, the Boating and Outdoor Festival will kick off for four days, featuring a myriad of outdoor products and, of course, lots of boats. Produced by the Michigan Boating Industries Association (MBIA) and now in its second year, the festival also includes exhibits of waterfront living, fine art, vacation destinations, fishing and outdoor recreation products such as ATV's, motorcycles, paddle sports, among others.
Because MBIA packages it as a festival rather than a boat show, it's also rich in entertainment drawing cards. Some highlights include: shows featuring Duma the Water Skiing Dog; a large Kids Zone boasting bounce houses, stilt walkers, face painters and featuring appearances by Nickelodeon's Dora the Explorer; the "Ultimate Playground," a demo riding experience being hosted by BRP using Sea-Doos, ATVs, SSVs and Roadsters; and, of course, beer gardens, live music and more.
Farther south, if it's anything nautical, it will likely be found at the Nautical Marketplace at St. Johns Yacht Harbor (Charleston) set for this Saturday and produced by the South Carolina Marine Association (SCMA). This single day event will new and pre-owned boats, accessories and equipment displays, nautical antiques and "newtiques," arts and crafts, among other displays. The event is free to the public. In addition, it will be followed by a BBQ and party for SCMA members including a silent auction with a portion of the proceeds going to the Harry Hampton Memorial Wildlife Fund.
Special events take on a different slant for MarineMax (Clearwater, FL). These are aimed squarely at its customer base. In fact, MarineMax cleverly utilizes some large community events to create special customer gatherings. For example, the upcoming Red Bull Flugtag will "land" on the downtown Tampa waterfront Oct. 7-9. MarineMax will gather owners to view this entertaining event that has teams with homemade, human-powered flying machines piloting them off a 30-foot high deck in hopes of achieving flight. They all ultimately splash into the harbor below but it's a great show!
Similarly, MarineMax will take advantage of two more events. (1) "Guavaween" is an annual Tampa Halloween celebration drawing more than 100,000 people, featuring a parade, costume contests and concerts with national and local talent. MarineMax calls it their Guavaween Getaways for customers and it's for all ages. (2) In November, MarineMax customers will head for the big RibFest 2011 on the St. Petersburg waterfront. It's a family event featuring national touring bands, entertainment and, o course, the sweet small of BBQ fills downtown St. Pete.
In these and other similar events, MarineMax uses the occasions to cultivate their customer relationships. And, it demonstrates that if a dealer just looks around at what's happening in the area, he doesn't have to create the events to have effective customer events.