text:
\section{Introduction}
\subsection{Background}\label{sec:Backround}
Data sets in simulation and signal-reconstruction applications easily reach sizes too large for a single computer's random access memory (RAM).
A reasonable grid size for such tasks like
    galactic density reconstructions~\cite{greiner.electron_density} or
    multi-frequency imaging in radio astronomy~\cite{junklewitz.resolve}
    is a cube with a side resolution of $2048$.
Such a cube contains $2048^3 \approx 8.6 \cdot 10^9$ voxels.
Storing a 64-bit double for every voxel therefore consumes $64~\mathrm{GiB}$.
In practice one has to handle several or even many instances of those arrays which ultimately prohibits the use of single shared memory machines.
Apart from merely holding the arrays' data in memory,
    parallelization is needed to process those huge arrays within reasonable time.
This applies to basic arithmetics like addition and multiplication as well as to complex operations like Fourier transformation and advanced linear algebra,
    e.g. operator inversions or singular value decompositions.
Thus parallelization is highly advisable for code projects that must be scaled to high resolutions.

To be specific, the initial purpose of \textsc{d2o}\ was to provide parallelization to the package for Numerical Information Field Theory (\textsc{NIFTy})\cite{nifty},
    which permits the abstract and efficient implementation of sophisticated signal processing methods.
Typically, those methods are so complex on their own that a \textsc{NIFTy}\ user should not need to bother with parallelization details in addition to that.
It turned out that providing a generic encapsulation for parallelization to \textsc{NIFTy}\ is not straightforward
    as the applications \textsc{NIFTy}\ is used for are highly diversified.
The challenge hereby is that, despite all their peculiarities, for those applications numerical efficiency is absolutely crucial.
Hence, for encapsulating the parallelization effort in \textsc{NIFTy}\ we needed an approach that is flexible enough to adapt to those different applications such that numerical efficiency can be preserved: \textsc{d2o}.

\textsc{d2o}\ is implemented in Python.
As a high-level language with a low-entry barrier Python is widely used in computational science.
It has a huge standard library and an active community for 3rd party packages.
For computationally demanding applications Python is predominantly used as a steering language for external compiled modules because Python itself is slow for numerics.

This article is structured as follows.
Section \ref{sec:aim} gives the aims of \textsc{d2o},
    and section \ref{sec:alternatives} describes alternative data distribution packages.
We dicuss the code architecture in section \ref{sec:code_arch} ,
    the basic usage of \textsc{d2o}\ in section \ref{sec:basic_usage},
    and the numerical scaling behavior in section \ref{sec:performance_scalability}.
Section \ref{sec:summary} contains our conclusion
    and appendix \ref{sec:adv_usage_func_behav} describes the detailed usage of \textsc{d2o}.

\subsection{Aim}\label{sec:aim}
As most scientists are not fully skilled software engineers, for them the hurdle for developing parallelized code is high.
Our goal is to provide data scientists with a numpy array-like object (cf.\ \emph{numpy}~\cite{walt.numpy}) that distributes data among several nodes of a cluster in a user-controllable way.
The user, however, shall not need to have profound knowledge about parallel programming with a system like \emph{MPI}~\cite{mpi-1-standard, mpi-2-standard} to achieve this.
The transition to use \emph{distributed\_data\_objects}\ instead of numpy arrays in existing code must be as straightforward as possible.
Hence, \textsc{d2o}\ shall in principle run -- at least in a non-parallelized manner -- with standard-library dependencies available;
    the packages needed for parallel usage should be easily available.
Whilst providing a global-minded interface, the node's local data should be directly accessible in order to enable the usage in specialized high-performance modules.
This approach matches with the theme of \emph{\mbox{DistArray}} \cite{distarray}: ``Think globally, act locally''.
Regarding \textsc{d2o}'s architecture we do not want to make any a-priori assumptions about the specific distribution strategy, but retain flexibility:
    it shall be possible to adapt to specific constraints induced from third-party libraries a user may incorporate.
For example, a library for fast Fourier transformations like \emph{FFTW}~\cite{Frigo:1999:FFT:301618.301661} may rely on a different data-distribution model than
    a package for linear algebra operations like \emph{ScaLAPACK}~\cite{slug}\footnote{FFTW distributes slices of data, while ScaLAPACK uses a block-cyclic distribution pattern.}.
In the same manner it shall not matter whether a new distribution scheme stores data redundantly or not, e.g. when a node is storing not only a distinct piece of a global array, but also its neighboring (ghost) cells \cite{DadoneGhostCellGrid}.

Our main focus is on rendering extremely costly computations possible in the first place;
    not on improving the speed of simple computations that can be done serially.
Although primarily geared towards \emph{weak scaling}, it turns out that \textsc{d2o}{} performs very well in \emph{strong-scaling} scenarios, too; see section \ref{sec:performance_scalability} for details.

\subsection{Alternative Packages}\label{sec:alternatives}
There are several alternatives to \textsc{d2o}. We discuss the differences to \textsc{d2o}\ and why the alternatives are not sufficient for our needs.

\subsubsection{DistArray}
\emph{DistArray}~\cite{distarray} is very mature and powerful.
Its approach is very similar to \textsc{d2o}:
    It mimics the interface of a multi dimensional numpy array while distributing the data among nodes in a cluster.
However, DistArray involves a design decision that makes it inapt for our purposes:
    it has a strict client-worker architecture.
DistArray either needs an \emph{ipython ipcluster}~\cite{PER-GRA:2007.ipython} as back end or must be run with two or more MPI processes.
The former must be started before an interactive ipython session is launched.
This at least complicates the workflow in the prototyping phase
    and at most is not practical for batch system based computing on a cluster.
The latter enforces tool-developers who build on top of \emph{DistArray} to demand that their code always is run parallelized.
Both scenarios conflict with our goal of minimal second order dependencies and maximal flexibility,
    cf.\ section \ref{sec:aim}.
Nevertheless, its theme also applies to \textsc{d2o}: ``Think globally, act locally''.

\subsubsection{scalapy (ScaLAPACK)}
\emph{scalapy} is a Python wrapper around \emph{ScaLAPACK}~\cite{slug}, which is ``a library of high-performance linear algebra routines for parallel distributed memory machines''~\cite{scalapack.online}.
The \verb|scalapy.DistributedMatrix| class essentially uses the routines from ScaLAPACK and therefore is limited to the functionality of that:
    two-dimensional arrays and very specific block-cyclic distribution strategies that optimize numerical efficiency in the context of linear algebra problems.
In contrast, we are interested in $n$-dimensional arrays whose distribution scheme shall be arbitrary in the first place.
Therefore scalapy is not extensive enough for us.

\subsubsection{petsc4py (PETSc)}
\emph{petsc4py} is a Python wrapper around \emph{PETSc}, which ``is a suite of data structures and routines for the scalable (parallel) solution of scientific applications modeled by partial differential equations''~\cite{petsc-web-page}.
Regarding distributed arrays its scope is as focused as scalapy to its certain problem domain --
    here: solving partial differential equations.
The class for distributed arrays \verb|petsc4py.PETSc.DMDA| is limited to one, two and three dimensions
    as PETSc uses a highly problem-fitted distribution scheme.
We in contrast need $n$-dimensional arrays with arbitrary distribution schemes.
Hence, petsc4py is not suitable for us.

\section{Code Architecture}\label{sec:code_arch}
\subsection{Choosing the Right Level of Parallelization}
\textsc{d2o}\ distributes numerical arrays over a cluster in order to parallelize and therefore to speed up operations on the arrays themselves.
An application that is built on top of \textsc{d2o}\ can profit from its fast array operations that may be performed on a cluster.
However, there are various approaches how to deploy an algorithm on a cluster and
    \textsc{d2o}\ implements only one of them.
In order to understand the design decisions of \textsc{d2o}\
    and its position respective to other packages, cf.\ section \ref{sec:alternatives},
    we will now discuss the general problem setting of parallelization and possible approaches for that.
Thereby we reenact the decision process which led to the characteristics \textsc{d2o}\ has today.\\

\subsubsection{Vertical \& Horizontal Scaling}
Suppose we want to solve an expensive numerical problem which involves operations on data arrays.
To reduce the computation time one can in principle do two things.
Either use a faster machine -- \emph{vertical scaling} --
    or use more than one machine -- \emph{horizontal scaling}.
Vertical scaling has the advantage that existing code does not need to be changed\footnote{This is true if scaling up does not involve a change of the processor architecture.},
    but in many cases this is not appropriate.
Maybe one already uses the fastest possible machine,
    scaling up is not affordable
    or even the fastest machine available is still too slow.
Because of this, we choose horizontal scaling.
\subsubsection{High- \& Low-Level Parallelization}
With horizontal scaling we again face two choices:
    high- and low-level parallelization.
With high-level parallelization, many replicas of the algorithm run simultaneously,
    potentially on multiple machines.
Each instance then works independently if possible,
    solving an isolated part of the global problem.
At the end, the individual results get collected and merged.
The python framework \emph{pathos}~\cite{mckerns.pathos} provides functionality for this kind of procedure.\\
An example of high-level parallelization is a sample generator which draws from a probability distribution.
Using high-level parallelization many instances of the generator produce their own samples,
    which involves very little communication overhead.
The sample production process itself, however, is not sped up.\\
In low-level parallelization, several nodes work together on one basic task at a time.
For the above sample generator,
    this means that all nodes work on the same sample at a time.
Hence, the time needed for producing individual samples is reduced;
    they are serially generated by the cluster as a whole.
\subsubsection{Downsides}
Both of these approaches have their drawbacks.
For high-level parallelization the algorithm itself must be parallelizable.
Every finite algorithm has a maximum degree of intrinsic parallelization\footnote{For the exemplary sample generator the maximum degree of parallelization is the total number of requested samples.}.
If this degree is lower than the desired number of processes then high-level parallelization reaches its limits.
This is particularly true for algorithms that cannot be parallelized by themselves, like iterative schemes.
Furthermore, there can be an additional complication:
    if the numerical problem deals with extremely large objects it may be the case that it is not at all solvable by one machine alone\footnote{In case of the sample generator this would be the case if even one sample would be too large for an individual machine's RAM.}.\\
Now let us consider low-level parallelization.
As stated above,
    we assume that the solution of the given numerical problem involves operations on data arrays.
Examples for those are unary\footnote{E.g. the positive, negative or absolute values of the array's individual elements or the maximum, minimum, median or mean of all its elements.},
    binary\footnote{E.g. the sum, difference or product of two data arrays.} or
    sorting operations,
    but also more advanced procedures like Fourier transformations or (other)
    linear algebra operations.
Theoretically, the absolute maximum degree of intrinsic parallelization for an array operation is equal to the array's number of elements.
For comparison, the problems we want to tackle involve at least $10^8$ elements
    but most of the TOP500~\cite{TOP500} supercomputers possess $10^6$ cores or less.
At first glance this seems promising.
But with an increasing number of nodes that participate in one operation the computational efficiency may decrease considerably.
This happens if the cost of the actual numerical operations becomes comparable to the generic program and inter-node communication overhead.
The ratios highly depend on the specific cluster hardware and the array operations performed.

\subsubsection{Problem Sizes}
Due to our background in signal reconstruction and grid-based simulations, we decide to use low-level parallelization for the following reasons.
First, we have to speed up problems that one cannot parallelize algorithmically,
    like fixed-point iterations or step-wise simulations.
Second, we want to scale our algorithms to higher resolutions
    while keeping the computing time at least constant.
Thereby the involved data arrays become so big that a single computer would be oversubscribed.
Because of this, the ratio of \emph{array size} to \emph{desired degree of parallelization} does not become such that the computational efficiency would decrease considerably.
In practice we experience a good scaling behavior with up to $\approx 10^3$ processes\footnote{This was the maximum number of processes available for testing.} for problems of size $8192^2$,
    cf.\ section \ref{sec:performance_scalability}.
Hence, for our applications the advantages of low-level parallelization clearly outweigh its drawbacks.

\subsection{\textsc{d2o}\ as Layer of Abstraction}
Compared to high-level parallelization,
    the low-level approach is more complicated to implement.
In the best case, for the former one simply runs the serial code in parallel on the individual machines;
    when finished one collects and combines the results.
For the latter, when doing the explicit coding one deals with local data portions of the global data array on the individual nodes of the cluster.
Hence, one has to keep track of additional information:
    for example, given a distribution scheme, which portion of the global data is stored on which node of the cluster?
Keeping the number of cluster nodes, the size and the dimensionality of the data arrays arbitrary implies a considerable complication for indexing purposes.
By this, while implementing an application one has to take care of two non-trivial tasks.
On the one hand, one must program the logic of distributing and collecting the data; i.e.\ the data handling.
On the other hand, one must implement the application's actual (abstract) algorithm.
Those two tasks are conceptually completely different and therefore a mixture of implementations should be avoided.
Otherwise there is the risk that features of an initial implementation -- like the data distribution scheme -- become hard-wired to the algorithm, inhibiting its further evolution.
Thus it makes sense to insert a layer of abstraction between the algorithm code and the data distribution logic.
Then the abstract algorithm can be written in a serial style from which all knowledge and methodology regarding the data distribution is encapsulated.
This layer of abstraction is \textsc{d2o}.

\subsection{Choosing a Parallelization Environment}
To make the application spectrum of \textsc{d2o}\ as wide as possible we want to maximize its portability and reduce its dependencies.
This implies that -- despite its parallel architecture -- \textsc{d2o}\ must just as well run within a single-process environment for cases when no elaborate parallelization back end is available.
But nevertheless, \textsc{d2o}\ must be massively scalable.
This relates to the question of which distribution environment should be used.
There are several alternatives:
\begin{itemize}
	\item Threading and multiprocessing:
        These two options limit the application to a single machine which conflicts with the aim of massive scalability.
	\item (py)Spark~\cite{Zaharia:2010:SCC:1863103.1863113} and hadoop~\cite{hadoop}: These modern frameworks are very powerful but regrettably too abstract for our purposes,
        as they prescind the location of individual portions of the full data.
    Building a numpy-like interface would be disproportionately hard or even unfeasible.
    In addition to that, implementing a low-level interface for highly optimized applications which interact with the node's local data  is not convenient within pySpark.
    Lastly, those frameworks are usually not installed as standard dependencies on scientific HPC clusters.
	\item MPI~\cite{mpi-1-standard, mpi-2-standard}:
    The \emph{Message Passing Interface} is available on virtually every HPC cluster via well-tested implementations like \emph{OpenMPI}~\cite{gabriel04:_open_mpi}, \emph{MPICH2}~\cite{mpich2} or \emph{\mbox{Intel MPI}}~\cite{intelmpi}.
    The open implementations are also available on commodity multicore hardware like desktops or laptops.
    A Python interface to MPI is given by the Python module \emph{mpi4py}~\cite{Dalcin20051108.mpi4py}.
    MPI furthermore offers the right level of abstraction for hands-on control of distribution strategies for the package developers.
\end{itemize}
Given these features we decide to use \emph{MPI} as the parallelization environment for \textsc{d2o}.
We stress that in order to fully utilize \textsc{d2o}{} on multiple cores, a user does not need to know how to program in MPI; it is only necessary to execute the program via MPI as shown in the example in section~\ref{sec:distr_arrays}.

\subsection{Internal Structure}\label{sec:internal_structure}
\subsubsection{Composed Object}\label{sec:composed_object}
A main goal for the design of \textsc{d2o}\ was to make no a-priori assumptions about the specific distribution strategies that will be used in order to spread array data across the nodes of a cluster.
Because of this, \textsc{d2o}'s distributed array -- \verb|d2o.distributed_data_object| -- is a composed object; cf.\ figure \ref{fig_object_structure}.\\
The \texttt{\detokenize{distributed_data_object}}\ itself provides a rich user interface,
    and makes sanity and consistency checks regarding the user input.
In addition to that, the \texttt{\detokenize{distributed_data_object}}\ possesses an attribute called \verb|data|.
Here the MPI processes' local portion of the global array data is stored,
    even though the \texttt{\detokenize{distributed_data_object}}\ itself will never make any assumptions about its specific content
    since the distribution strategy is arbitrary in the first place.
The \texttt{\detokenize{distributed_data_object}}{} is the only object of the \textsc{d2o}{} library that a casual user would interact with.\\
For all tasks that require knowledge about the certain distribution strategy every \texttt{\detokenize{distributed_data_object}}\ possesses an instance of a \verb|d2o.distributor| subclass.
This object stores all the distribution-scheme and cluster related information
    it needs in order to scatter (gather) data to (from) the nodes and to serve for special methods,
    e.g.\ the array-cumulative sum.
The \texttt{\detokenize{distributed_data_object}}{} builds its rich user interface on top of those abstracted methods of its distributor.\\
The benefit of this strict separation is that the user interface becomes fully detached from the distribution strategy;
    may it be block-cyclic or slicing,
    or have neighbor ghost cells or not, et cetera.
Currently there are two fundamental distributors available:
    a generic \emph{slicing}-\footnote{The slicing is done along the first array axis.} and a \emph{not}-distributor.
From the former, three special slicing distributors are derived:
    \emph{fftw}\footnote{The fftw-distributor uses routines from the pyFFTW~\cite{pyfftw, Frigo:1999:FFT:301618.301661} package~\cite{Frigo:1999:FFT:301618.301661} for the data partitioning.},
    \emph{equal}\footnote{The equal-distributor tries to split the data in preferably equal-sized parts.} and
    \emph{freeform}\footnote{The local data array's first axis is of arbitrary length for the freeform-distributor.}.
The latter, the \emph{not}-distributor, does not do any data-distribution or -collection but stores the full data on every node redundantly.

\subsubsection{Advantages of a Global View Interface}
\textsc{d2o}'s global view interface makes it possible to build software that remains completely independent from the distribution strategy and the used number of cluster processes.
This in turn enables the development of 3rd party libraries that are very end-use-case independent.
An example for this may be a mathematical optimizer;
    an object which tries to find for a given scalar function $f$ an input vector $\vec{x}$ such that the output $y = f(\vec{x})$ becomes minimal.
It is interesting to note that many optimization algorithms solely use basic arithmetics like vector addition or scalar multiplication when acting on $\vec{x}$.
As such operations act locally on the elements of an array,
    there is no preference for one distribution scheme over another
    when distributing $\vec{x}$ among nodes in a cluster.
Two different distribution schemes will yield the same performance
    if their load-balancing is on a par with each other.
Further assume that $f$ is built on \textsc{d2o}, too.
On this basis, one could now build an application that uses the minimizer
    but indeed has a preference for a certain distribution scheme.
This may be the case if the load-balancing of the used operations is non-trivial
    and therefore only a certain distribution scheme guarantees high evaluation speeds.
While the application's developer therefore enforces this scheme,
    the minimizer remains completely unaffected by this as it is agnostic of the array's distribution strategy.

\section{Basic Usage}\label{sec:basic_usage}
In the subsequent sections we will illustrate the basic usage of \textsc{d2o}{} in order to explain its functionality and behavior.
A more extended discussion is given in Appendix \ref{sec:adv_usage_func_behav}.
Our naming conventions are:
\begin{itemize}
	\item instances of the \verb|numpy.ndarray| class are labeled \verb|a| and \verb|b|,
	\item instances of \verb|d2o.distributed_data_object| are labeled \verb|obj| and \verb|p|.
\end{itemize}
In addition to these examples, the interested reader is encouraged to have a look into the \texttt{\detokenize{distributed_data_object}}{} method's docstrings for further information; cf. the project's web page \url{https://gitlab.mpcdf.mpg.de/ift/D2O}.\\

\subsection{Initialization}
Here we discuss how to initialize a \texttt{\detokenize{distributed_data_object}}{} and compare some of its basic functionality to that of a \verb|numpy.ndarray|.
First we import the packages.
\begin{lstlisting}[language=iPython]
	|\refstepcounter{inCounter}{\color{lightgreen}In [{\color{green}\arabic{inCounter}}]:}| import numpy as np
	|\refstepcounter{inCounter}{\color{lightgreen}In [{\color{green}\arabic{inCounter}}]:}| from d2o import distributed_data_object
\end{lstlisting}
Now we set up some test data using numpy.
\begin{lstlisting}[language=iPython]
    |\refstepcounter{inCounter}{\color{lightgreen}In [{\color{green}\arabic{inCounter}}]:}| a = np.arange(12).reshape((3, 4))
    |\refstepcounter{inCounter}{\color{lightgreen}In [{\color{green}\arabic{inCounter}}]:}| a
    |{\color{lightred}Out[{\color{red}\arabic{inCounter}}]:}| array([[ 0,  1,  2,  3],
                  [ 4,  5,  6,  7],
                  [ 8,  9, 10, 11]])
\end{lstlisting}
One way to initialize a \texttt{\detokenize{distributed_data_object}}{} is to pass an existing numpy array.
\begin{lstlisting}[language=iPython]
    |\refstepcounter{inCounter}{\color{lightgreen}In [{\color{green}\arabic{inCounter}}]:}| obj = distributed_data_object(a)
    |\refstepcounter{inCounter}{\color{lightgreen}In [{\color{green}\arabic{inCounter}}]:}| obj
    |{\color{lightred}Out[{\color{red}\arabic{inCounter}}]:}| <distributed_data_object>
           array([[ 0,  1,  2,  3],
                   [ 4,  5,  6,  7],
                   [ 8,  9, 10, 11]])
\end{lstlisting}
The output of the \verb|obj| call shows the local portion of the global data available in this process.

\subsection{Arithmetics}
Simple arithmetics and point-wise comparison work as expected from a numpy array.
\begin{lstlisting}[language=iPython]
    |\refstepcounter{inCounter}{\color{lightgreen}In [{\color{green}\arabic{inCounter}}]:}| (2*obj, obj**3, obj >= 5)
    |{\color{lightred}Out[{\color{red}\arabic{inCounter}}]:}| (<distributed_data_object>
            array([[ 0,  2,  4,  6],
                   [ 8, 10, 12, 14],
                   [16, 18, 20, 22]]),
            <distributed_data_object>
            array([[   0,    1,    8,   27],
                   [  64,  125,  216,  343],
                   [ 512,  729, 1000, 1331]]),
            <distributed_data_object>
            array([[False, False, False, False],
                   [False,  True,  True,  True],
                   [ True,  True,  True,  True]], dtype=bool))
\end{lstlisting}
Please note that the \texttt{\detokenize{distributed_data_object}}{} tries to avoid inter-process communication whenever possible.
Therefore the returned objects of those arithmetic operations are instances of \texttt{\detokenize{distributed_data_object}}{}, too.
However, the \textsc{d2o}{} user must be careful when combining \emph{distributed\_data\_objects}{} with numpy arrays.
If one combines two objects with a binary operator in Python (like \verb|+|, \verb|-|, \verb|*|, \verb|\|, \verb
    it will try to call the respective method (\verb|__add__|, \verb| __sub__|, etc...) of the \emph{first} object.
If this fails, i.e. if it throws an exception,
    Python will try to call the \emph{reverse} methods of the \emph{second} object (\verb|__radd__|, \verb|__rsub__|, etc...):
\begin{lstlisting}[language=iPython]
    |\refstepcounter{inCounter}{\color{lightgreen}In [{\color{green}\arabic{inCounter}}]:}| a + 1; # calls a.__add__(1) -> returns a numpy array
    |\refstepcounter{inCounter}{\color{lightgreen}In [{\color{green}\arabic{inCounter}}]:}| 1 + a; # 1.__add__ not existing -> a.__radd__(1)
\end{lstlisting}
Depending on the conjunction's ordering, the return type may vary when combining numpy arrays with \emph{distributed\_data\_objects}{}.
If the numpy array is in the first place,
    numpy will try to extract the second object's array data using its \verb|__array__| method.
This invokes the \texttt{\detokenize{distributed_data_object}}'s \verb|get_full_data| method that communicates the full data to every process.
For large arrays this is extremely inefficient and should be avoided by all means.
Hence, it is crucial for performance to assure that the \texttt{\detokenize{distributed_data_object}}'s methods will be called by Python.
In this case, the locally relevant parts of the array are extracted from the numpy array and then efficiently processed as a whole.
\begin{lstlisting}[language=iPython]
    |\refstepcounter{inCounter}{\color{lightgreen}In [{\color{green}\arabic{inCounter}}]:}| a + obj # numpy converts obj -> inefficient
    |{\color{lightred}Out[{\color{red}\arabic{inCounter}}]:}| array([[ 0,  2,  4,  6], # note: numpy.ndarray
                  [ 8, 10, 12, 14],
                  [16, 18, 20, 22]])

    |\refstepcounter{inCounter}{\color{lightgreen}In [{\color{green}\arabic{inCounter}}]:}| obj + a # obj processes a -> efficient
    |{\color{lightred}Out[{\color{red}\arabic{inCounter}}]:}| <distributed_data_object>
            array([[ 0,  2,  4,  6],
                   [ 8, 10, 12, 14],
                   [16, 18, 20, 22]])
\end{lstlisting}
\subsection{Array Indexing}
The \texttt{\detokenize{distributed_data_object}}{} supports most of numpy's indexing functionality,
    so it is possible to work with scalars, tuples, lists, numpy arrays and \emph{distributed\_data\_objects}{} as input data.
Every process will extract its locally relevant part of the given data-object and then store it;
    cf.\ \ref{sec:advanced_indexing}.
\begin{lstlisting}[language=iPython]
    |\refstepcounter{inCounter}{\color{lightgreen}In [{\color{green}\arabic{inCounter}}]:}| obj
    |{\color{lightred}Out[{\color{red}\arabic{inCounter}}]:}| <distributed_data_object>
            array([[ 0,  1,  2,  3],
                   [ 4,  5,  6,  7],
                   [ 8,  9, 10, 11]])

    |\refstepcounter{inCounter}{\color{lightgreen}In [{\color{green}\arabic{inCounter}}]:}| obj[1] # extract a row
    |{\color{lightred}Out[{\color{red}\arabic{inCounter}}]:}| <distributed_data_object>
            array([4, 5, 6, 7])

    |\refstepcounter{inCounter}{\color{lightgreen}In [{\color{green}\arabic{inCounter}}]:}| obj[1,-2] # extract single entry
    |{\color{lightred}Out[{\color{red}\arabic{inCounter}}]:}| 6

    |\refstepcounter{inCounter}{\color{lightgreen}In [{\color{green}\arabic{inCounter}}]:}| obj[::2, 1::2] # slicing notation
    |{\color{lightred}Out[{\color{red}\arabic{inCounter}}]:}| <distributed_data_object>
            array([[ 1,  3],
                   [ 9, 11]])

    # sets data using slicing
    |\refstepcounter{inCounter}{\color{lightgreen}In [{\color{green}\arabic{inCounter}}]:}| obj[::2, 1::2] = [[111, 222], [333, 444]]
    |\refstepcounter{inCounter}{\color{lightgreen}In [{\color{green}\arabic{inCounter}}]:}| obj
    |{\color{lightred}Out[{\color{red}\arabic{inCounter}}]:}| <distributed_data_object>
            array([[  0, 111,   2, 222],
                   [  4,   5,   6,   7],
                   [  8, 333,  10, 444]])
\end{lstlisting}
By default it is assumed that all processes use the \emph{same} key-object when accessing data.
See section \ref{sec:local_keys} for more details regarding process-individual indexing.

\subsection{Distribution Strategies}
In order to specify the distribution strategy explicitly one may use the ``\verb|distribution_strategy|'' keyword:
\begin{lstlisting}[language=iPython]
    |\refstepcounter{inCounter}{\color{lightgreen}In [{\color{green}\arabic{inCounter}}]:}| obj = distributed_data_object(
                    a, distribution_strategy='equal')
    |\refstepcounter{inCounter}{\color{lightgreen}In [{\color{green}\arabic{inCounter}}]:}| obj.distribution_strategy
    |{\color{lightred}Out[{\color{red}\arabic{inCounter}}]:}| 'equal'
\end{lstlisting}
See section \ref{sec:distribution_strategies} for more information on distribution strategies.

\subsection{Distributed Arrays} \label{sec:distr_arrays}
To use \textsc{d2o}{} in a distributed manner, one has to create an MPI job.
This example shows how four MPI processes hold individual parts of the global data and how distributed read \& write access works.
The script is started via the command:
\begin{verbatim}
mpirun -n 4 python get_set_data.py
\end{verbatim}

\begin{lstlisting}[language=iPython]
    # get_set_data.py
    from mpi4py import MPI
    import numpy as np
    from d2o import distributed_data_object
    # Get the process' rank number (0,1,2,3) from MPI
    rank = MPI.COMM_WORLD.rank

    # Initialize some data
    a = np.arange(16).reshape((4,4))
    # Initialize the distributed_data_object
    obj = distributed_data_object(a) |\label{listing_dist_arrays_1}|

    # Print the process' local data
    print (rank, obj.get_local_data()) |\label{listing_dist_arrays_2}|
    # extract data via slicing
    print (rank, obj[0:3:2, 1:3].get_local_data()) |\label{listing_dist_arrays_3}|

    b = -np.arange(4).reshape((2,2))
    obj[2:4,1:3] = b # Write b into obj |\label{listing_dist_arrays_4}|

    # Print the process' local data
    print (rank, obj.get_local_data())

    # Consolidate the data
    full_data = obj.get_full_data() |\label{listing_dist_arrays_5}|
    if rank == 0: print (rank, full_data)
\end{lstlisting}
The \texttt{\detokenize{distributed_data_object}}{} gets initialized in line \ref{listing_dist_arrays_1} with the following array:
\begin{verbatim}
array([[ 0,  1,  2,  3],
       [ 4,  5,  6,  7],
       [ 8,  9, 10, 11],
       [12, 13, 14, 15]]))
\end{verbatim}
Here, the script is run in four MPI processes; cf.\ \verb|mpirun -n 4 [...]|.
The data is split along the first axis; the print statement in line \ref{listing_dist_arrays_2} yields the four pieces:
\begin{verbatim}
(0, array([[ 0,  1,  2,  3]]))
(1, array([[ 4,  5,  6,  7]]))
(2, array([[ 8,  9, 10, 11]]))
(3, array([[12, 13, 14, 15]]))
\end{verbatim}
The second print statement (line \ref{listing_dist_arrays_3}) illustrates the behavior of data extraction;
    \verb|obj[0:3:2, 1:3]| is slicing notation for the entries 1, 2, 9 and 10\footnote{This notation can be decoded as follows. The numbers in a slice correspond to \verb|start:stop:step| with \verb|stop| being exclusive. \verb|obj[0:3:2, 1:3]| means to take every second line from the lines 0, 1 and 2, and to then take from this the elements in collumns 1 and 2.}.
This expression returns a \texttt{\detokenize{distributed_data_object}}{} where the processes possess the individual portion of the requested data.
This means that the distribution strategy of the new (sub-)array is determined by and aligned to that of the original array.
\begin{verbatim}
(0, array([[1, 2]]))
(1, array([], shape=(0, 2), dtype=int64)) # empty
(2, array([[ 9, 10]]))
(3, array([], shape=(0, 2), dtype=int64)) # empty
\end{verbatim}
The result is a \texttt{\detokenize{distributed_data_object}}\ where the processes 1 and 3 do not possess any data as they had no data to contribute to the slice in \verb|obj[0:3:2, 1:3]|.
In line \ref{listing_dist_arrays_4} we store a small 2x2 block \verb|b| in the lower middle of \verb|obj|. The process' local data reads:
\begin{verbatim}
(0, array([[ 0,  1,  2,  3]]))
(1, array([[ 4,  5,  6,  7]]))
(2, array([[ 8,  0, -1, 11]]))
(3, array([[12, -2, -3, 15]]))
\end{verbatim}
Finally, in line \ref{listing_dist_arrays_5} we use \verb|obj.get_full_data()| in order to consolidate the distributed data;
    i.e. to communicate the individual pieces between the processes and merge them into a single numpy array.
\begin{verbatim}
(0, array([[ 0,  1,  2,  3],
           [ 4,  5,  6,  7],
           [ 8,  0, -1, 11],
           [12, -2, -3, 15]]))
\end{verbatim}

\section{Performance and Scalability}\label{sec:performance_scalability}
In this section we examine the scaling behavior of a \texttt{\detokenize{distributed_data_object}}\ that uses the \emph{equal} distribution strategy.
The timing measurements were performed on the C2PAP Computing Cluster~\cite{c2pap}\footnote{The C2PAP computing cluster consists of 128 nodes, each possessing two Intel Xeon CPU E5-2680 (8 cores \@ 2.70~GHz + hyper-threading, 64~KiB L1 per core, 256~KiB L2 cache per core, 20~MiB L3 cache shared for all 8 cores) and 64 GiB RAM each. The nodes are connected via Mellanox Infiniband 40 Gbits/s}.
The software stack was built upon \emph{Intel MPI 5.1}, \emph{mpi4py 2.0}, \emph{numpy 1.11} and \emph{python 2.7.11}.
For measuring the individual timings we used the Python standard library module \emph{timeit} with a fixed number of $100$ repetitions.

Please note that \textsc{d2o}\ comes with an extensive suite of unit tests featuring a high code-coverage rate.
By this we assure \textsc{d2o}'s continuous code development to be very robust
    and provide the users with a reliable interface definition.

Regarding \textsc{d2o}'s performance there are two important scaling directions:
    the size of the array data and
    the number of MPI processes.
One may distinguish three different contributions to the overall computational cost.
First, there is data management effort that every process has to cover itself.
Second, there are the costs for inter-MPI-process communication.
And third, there are the actual numerical costs.

\textsc{d2o}\ has size-independent management overhead compared to numpy.
Hence, the larger the arrays are for which \textsc{d2o}\ is used, the more efficient the computations become.
We will see below that there is a certain array size per node
    -- roughly $2^{16}$ elements --
    from which on d2o's management overhead becomes negligible compared to the purely numerical costs.
This size corresponds to a two-dimensional grid with a resolution of $256 \times 256$ or equivalently $0.5~\mathrm{MiB}$ of 64-bit doubles.
In section \ref{sec:scale_size} we focus on this very ratio of management overhead to numerical costs.

\textsc{d2o}\ raises the claim to be able to operate well running with a single process as well as in a highly parallelized regime.
In section \ref{sec:scale_nodes}, the scaling analysis regarding the MPI process count is done with a fixed local array size for which the process overhead is negligible compared to the numerical costs.
For this weak scaling analysis we are interested in the costs arising from inter-process communication compared to those of actual numerics.

In the following three sections, we study the strong scaling of \textsc{d2o}{},
    where the performance is a result of the combination of all three cost contributions.
Section \ref{sec:strong_scaling} covers the case in which the number of MPI processes is increased while the array size is left constant.
In section \ref{sec:comp_distarray} we compare \textsc{d2o}'s to \emph{DistArray}'s\cite{distarray} performance
and finally, in section \ref{sec:wiener} we benchmark \textsc{d2o}'s strong-scaling behavior when applied to a real-world application:
    a Wiener filter signal reconstruction.

A discussion on \textsc{d2o}'s efficient Python iterators can be found in appendix \ref{sec:perf_iter}.

\subsection{Scaling the Array Size}\label{sec:scale_size}
One may think of \textsc{d2o}\ as a layer of abstraction that is added to numpy arrays
    in order to take care of data distribution and collection among multiple MPI processes.
This abstraction comes with inherent Python overhead, separately for each MPI process.
Therefore, if one wants to analyze how the ratio of \emph{management overhead} to \emph{actual numerical effort} varies with the data size,
    only the individual process' data size is important.
Because of this, all timing tests for this subsection were carried out with one MPI process only.

A common task during almost all numerical operations is to create a new array object for storing its results\footnote{Exceptions to this are inplace operations which reuse the input array for the output data.}.
Hence, the speed of object creation can be crucial for overall performance there.
Note that in contrast to a numpy array which basically just allocates RAM,
    several things must be done during the initialization of a \texttt{\detokenize{distributed_data_object}}.
The Python object instance itself must be created,
    a distributor must be initialized which involves parsing of user input,
    RAM must be allocated,
    the \texttt{\detokenize{distributed_data_object}}\ must be registered with the d2o\_librarian (cf.\ \ref{sec:librarian}),
    and, if activated, inter-MPI-process communication must be done for performing sanity checks on the user input.

By default the initialization requires \mus{60} to complete for a \texttt{\detokenize{distributed_data_object}}\ with a shape of \verb|(1,)| when run within one single MPI process.
Using this trivial shape makes the costs for memory allocation negligible compared to the others tasks.
Hence, those \mus{60} represent \textsc{d2o}'s constant overhead compared to numpy,
    since a comparable numpy array requires $\approx \mus{0.4}$ for initialization.

In order to speed up the initialization process one may disable all sanity checks on the user input that require MPI communication,
    e.g. if the same datatype was specified in all MPI processes.
Even when run with one single process,
    skipping those checks reduces the costs by \mus{27} from \mus{60} to \mus{33}.

Because of the high costs, it is important to avoid building \emph{distributed\_data\_objects}\ from scratch over and over again.
A simple measure against this is to use inplace operations like \verb|obj += 1| instead of \verb|obj = obj + 1| whenever possible.
This is generally a favorable thing to do -- also for numpy arrays --
    as this saves the costs for repeated memory allocation.
Nonetheless, also non-inplace operations can be improved in many cases,
    as often the produced and the initial \texttt{\detokenize{distributed_data_object}}\ have all of their attributes in common, except for their data:
    they are of the same shape and datatype, and use the same distribution strategy and MPI communicator;
    cf.\ \verb|p = obj + 1|.
With \verb|obj.copy()| and \verb|obj.copy_empty()| there exist two cloning methods that we implemented to be as fast as allowed by pure Python.
Those methods reuse as much already initialized components as possible and are therefore faster than a fresh initialization:
    for the \texttt{\detokenize{distributed_data_object}}\ from above
    \verb|obj.copy()| and \verb|obj.copy_empty()| consume \mus{7.9} and \mus{4.3}, respectively.

Table \ref{table_scaling_size} shows the performance ratio in percent between serial \textsc{d2o}\ and numpy.
The array sizes range from $2^0=1$ to $2^{25} \approx 3.3\cdot10^7$ elements.
In the table, $100\%$ would mean that \textsc{d2o}\ is as fast as numpy.

The previous section already suggested that for tasks that primarily consist of initialization work -- like \emph{array creation} or \verb|copy_empty| -- \textsc{d2o}\ will clearly follow behind numpy.
However, increasing the array size from $2^{20}$ to $2^{22}$ elements implies a considerable performance drop for numpy's memory allocation.
This in turn means that for arrays with more than $2^{22}$ elements \textsc{d2o}'s relative overhead becomes less significant: e.g.\ \verb|np.copy_empty| is then only a factor of four faster than \verb|obj.copy_empty()|.

Functions like \verb|max| and \verb|sum| return a scalar number;
    no expensive return-array must be created.
Hence, \textsc{d2o}'s overhead is quite modest:
    even for size $1$ arrays, \textsc{d2o}'s relative performance lies above $50\%$.
Once the size is greater than $2^{18}$ elements the performance is higher than $95\%$.

\verb|obj[::-2]| is \emph{slicing syntax} for \emph{``take every second element from the array in reverse order''}.
It illustrates the costs of the data-distribution and collection logic
    that even plays a significant role if there is no inter-process communication involved.
Again, with a large-enough array size, \textsc{d2o}'s efficiency becomes comparable to that of numpy.

Similarly to \verb|obj[::-2]|, the remaining functions in the table return a \texttt{\detokenize{distributed_data_object}}{} as their result and therefore suffer from its initialization costs.
However, with an array size of $2^{16}$ elements and larger \textsc{d2o}'s relative performance is at least greater than approximately $65\%$.

An interesting phenomenon can be observed for \verb|obj + 0| and \verb|obj + obj|:
As for the other functions, their relative performance starts to increase significantly when an array size of $2^{16}$ is reached.
However, in contrast to \verb|obj += obj| which then immediately scales up above $95\%$,
    the relative performance of the non-inplace additions temporarily \emph{decreases} with increasing array size.
This may be due to the fact that given our test scenario $2^{18}$ elements almost take up half of the cache of C2PAP's Intel E5-2680 CPUs.
\textsc{d2o}'s memory overhead is now responsible for the fact, that its non-inplace operations -- which need twice the initial memory -- cannot profit that strongly from the cache anymore,
    whereas the numpy array still operates fast.
Once the array size is above $2^{22}$ elements numpy's just as \textsc{d2o}'s array-object is too large for profiting from the cache and therefore become comparably fast again:
    the relative performance is then greater than $98\%$.

Thus, when run within a single process, \textsc{d2o}\ is ideally used for arrays larger than $2^{16}=65536$ elements which corresponds to $512~\mathrm{KiB}$.
From there the management overhead becomes less significant than the actual numerical costs.

\subsection{Weak Scaling: Proportional Number of Processes and Size of Data}\label{sec:scale_nodes}
Now we analyze the scaling behavior of \textsc{d2o}\ when run with several MPI processes.
Repeating the beginning of section \ref{sec:performance_scalability}, there are three contributions to the execution time.
First, the fixed management overhead that every process has to cover itself, second, the communication overhead and third, the actual numerical costs.
In order to filter out the effect of a changing contribution of management overhead, in this section we fix the MPI processes' local array size to a fixed value.
Hence, now the global data size is proportional to the number of MPI processes.

Table \ref{table_scaling_nodes} shows the performance of various array operations normalized to the time \textsc{d2o}\ needs when running with one process only.
Assuming that \textsc{d2o}\ had no communication overhead and an operation scaled perfectly linearly with array size, the performance would be rated at $100\%$.

In theory, operations that do not inherently require inter-process communication like point-wise array addition or subtraction ideally scale linearly.
And in fact,
    \textsc{d2o}\ scales excellently with the number of processes involved for those functions:
    here we tested \verb|copy|, \verb|copy_empty|, \verb|sum(axis=1)|, \verb|obj + 0|, \verb|obj + obj|, \verb|obj += obj| and \verb|sqrt|.

Comparing \verb|sum(axis=0)| with \verb|sum(axis=1)| illustrates the performance difference between those operations that involve inter-process communication and those that don't:
    the \emph{equal} distribution strategy slices the global array along its first axis in order to distribute the data among the individual MPI processes.
Hence, \verb|sum(axis=0)| -- which means to take the sum along the first axis -- does intrinsically involve inter-process communication
    whereas \verb|sum(axis=1)| does not.
Similarly to \verb|sum(axis=0)| also the remaining functions in table \ref{table_scaling_nodes} are affected by an increasing number of processes as they involve inter-process communication.

But still, even if -- for example in case of \verb|sum(axis=0)| -- the relative performance may drop to $28.2\%$ when using 256 processes,
    this means that the operation just took 3.5 times longer than the single-process run,
    whereat the array size has been increased by a factor of $256$.
This corresponds to a speedup factor of $72.2$.

\subsection{Strong Scaling: Varying Number of Processes with a Fixed Size of Data}\label{sec:strong_scaling}
Similarly to the previous section \ref{sec:scale_nodes}, we vary the number of processes but now fix the data size \emph{globally} instead of locally.
This corresponds to the real-life scenario in which the problem size and resolution are already set -- maybe by environmental conditions --
    and now one tries to reduce the run time by using more cores.
Since the size of the local data varies with the number of processes,
    the overall scaling behavior is now a mixture of the varying ratio between management overhead and process-individual numerical costs,
    and the fact that an increasing amount of CPU cache becomes available at the expense of increased communication effort.
Table \ref{table_scaling_nodes_strong} shows the benchmarking results for the same set of operations as used in the previous section on weak scaling
    and the results are reasonable.
Those operations in the list that inherently cannot scale strongly as they consist of purely node-individual work, namely the \verb|initialization| and \verb|copy_empty|, show that their performance just does not increase with the number of processes.
In contrast, operations without communication effort benefit strongly from the increasing total amount of CPU cache combined with smaller local arrays;
    above all \verb|copy| which is about 3 times faster than what one would expect from linear scaling  to 256 processes\footnote{This very strong scaling is indeed realistic: when analyzing pure numpy arrays one gets speedups in the order of magnitude of even $800~\%$.}.

In theory, the strong-scaling behavior is the combination of the size- and weak-scaling we discussed in sections \ref{sec:scale_size} and \ref{sec:scale_nodes}.
In order to verify whether the strong-scaling behavior makes sense, we estimate the strong-scaling performance using the information from size- and weak-scaling.

We choose the \verb|sum()| method as our test case.
During the reduction phase, the $n$ MPI-processes exchange their local results with each other.
This corresponds to adding $n$ times a fixed amount of communication time to the total computing time needed.
Hence, we perform a linear fit to the weak-scaling data; cf.\ table~\ref{table_scaling_nodes}.
Furthermore, we assume that the local computation time is roughly proportional to the local-array size.
This is true for sufficiently large array sizes,
    since then numpy scales linearly
    and \textsc{d2o}\ is in a good efficiency regime, cf.\ \ref{table_scaling_size}.
Again we performed a linear fit but now on the size-scaling timing data.
Combining those two linear fits leads to the following run-time formula for applying \verb|sum()| to an array with shape $(4096, 4096)$:
\begin{equation}
t(n) = (0.0065n + 1.57/n)~\mathrm{s}
\end{equation}
In the case of linear scaling, $t(n)$ is expected to be equal to $t(1)/n$.
Hence, the relative performance $p(n)$ is the ratio between the two:
\begin{equation}
p(n)_{\mathrm{estimated}} = \frac{\nicefrac{t(1)}{n}}{t(n)} = \frac{241.8}{240.8 + n^2}
\end{equation}
Comparing the estimate with the actually measured relative performance -- cf.\ table \ref{table_scaling_nodes_strong} -- table \ref{tab_strong_scaling_estimate} shows that even under those rough assumptions the strong scaling behavior of \verb|sum()| can be explained as the combination of size- and weak scaling within about 20 \% accuracy.

\begin{table}[h!]
\caption{Strong Scaling Behavior: Estimate vs.\ Measurement}\label{tab_strong_scaling_estimate}
      \begin{tabular}{c|cccccccc}
            $\#\mathrm{processes:~} n$ & 1  & 4 & 8 & 16 & 32 & 64 & 128 & 256 \\ \hline
            $p(n)_{\mathrm{estimated}}$& 100 \% & 94.2 \% & 79.3 \% & 48.7 \% & 19.1 \% & 5.58 \% & 1.45 \% & 0.37 \% \\
            $p(n)_{\mathrm{measured}}$ & 100 \% & 91.7 \% & 74.1 \% & 60.9 \% & 24.4 \% & 7.05 \% & 1.82 \% & 0.45 \% \\
      \end{tabular}
\end{table}

\subsection{Strong Scaling: Comparison with DistArray}\label{sec:comp_distarray}
In section \ref{sec:alternatives} we discussed several competitors to \textsc{d2o}{}.
Because of their similarities, we conducted the strong scaling tests -- as far as possible\footnote{DistArray does, for example, not support negative step-sizes for slicing (\texttt{[::-2]}) and also the special method \texttt{bincount} is not available.} -- also with DistArray and compare the performance.
In table \ref{table_scaling_nodes_distarray} the results are shown for the subset of all operations from the previous sections that where available for DistArray, too.

While being at least on a par for numerical operations when being run single-threaded,
    \textsc{d2o}\ outperforms DistArray more and more with an increasing number of processes.
Furthermore, it is conspicuous that DistArray does not seem to support inplace array operations.
Because of this, the inplace addition \verb|obj+=obj| is way slower with DistArray than with \textsc{d2o}\ which is on a par with numpy in most cases, cf. tables \ref{table_scaling_size}, \ref{table_scaling_nodes} and \ref{table_scaling_nodes_strong}.

The fact that \textsc{d2o}\ is way more efficient when doing numerics on very small arrays -- like \verb|obj+0| using 256 processes -- indicates that \textsc{d2o}'s organizatorial overhead is much smaller than that of DistArray.
Supporting evidence for this is that the initialization of an empty DistArray (\verb|copy_empty|) becomes disproportionately costly when increasing the number of processes used.

\subsection{Strong Scaling: Real-World Application Speedup -- the Wiener filter}\label{sec:wiener}
\textsc{d2o}\ was initially developed for \textsc{NIFTy}\cite{nifty},
    a library for building signal inference algorithms in the framework of \emph{information field theory} (IFT)\cite{Ensslin:2008iu}.
Within \textsc{NIFTy}~v2 all of the parallelization is done via \textsc{d2o};
    the code of \textsc{NIFTy}\ itself is almost completely agnostic of the parallelization and completely agnostic of MPI.\\
A standard computational operation in IFT-based signal reconstruction is the \emph{Wiener filter}\cite{wiener1949}.
For this performance test, we use a Wiener filter implemented in \textsc{NIFTy}\ to reconstruct the realization of a Gaussian random field -- called the \emph{signal}.
Assume we performed a hypothetical measurement which produced some \emph{data}.
The data model is
\begin{equation}
\mathrm{data} = R(\mathrm{signal}) + \mathrm{noise}
\end{equation}
where $R$ is a smoothing operator and \emph{noise} is additive Gaussian noise.
In figure \ref{fig_wiener_filter} one sees the three steps:
\begin{itemize}
    \item the true signal we try to reconstruct,
    \item the data one gets from the hypothetical measurement, and
    \item the reconstructed signal field that according to the Wiener filter has most likely produced the $data$.
\end{itemize}
Table \ref{tab_wiener_filter} shows the scaling behavior of the reconstruction code,
    run with a resolution of $8192 \times 8192$.
Here, $n$ is the number of used processes and $t_n$ the respective execution time.
The relative speedup $s_n = 2t_2/t_n$\footnote{Since the combination of NIFTy and pyfftw exhibits an unexpected speed malus for one process, we chose the two-process timing as the benchmark's baseline.}
    is the ratio of execution times: parallel versus serial.
In the case of exactly linear scaling $s_n$ is equal to $n$.
Furthermore we define the scaling quality $q = 1/(1+\log(n/s_n))$,
    which compares $s_n$ with linear scaling in terms of orders of magnitude.
A value $q=1$ represents linear scaling and $q \ge 1$ super-linear scaling.
\begin{table}[h!]
\caption{Execution time scaling of a Wiener filter reconstruction on a grid of size $8192 \times 8192$.}\label{tab_wiener_filter}
      \begin{tabular}{c|ccccccccc}
            $\#\mathrm{nodes}$ & 1  & 1  & 1 & 1 & 2 & 4 & 8 & 16 & 32 \\
            $\#\mathrm{processes:~} n$ & 1  & 2  & 3 & 4 & 8 & 16 & 32 & 64 & 128 \\ \hline
            $t [\SI{}{\second}]$ & 1618 & 622.0 & 404.2 & 364.2 & 181.7 & 94.50 & 46.79 & 18.74 & 8.56 \\
            $s_n = 2t_2/t_n$ & 0.769 &  2.00 &  3.08 &  3.42 &  6.85 &  13.2 &  26.6 &  66.4 &  145\\
            $q_n = \scriptstyle{1/(1+\log(\frac{n}{s_n}))}$ & 0.900 &  1.00 &  1.01 &  0.94 &  0.94 &  0.92 &  0.93 &  1.02 &  1.06
      \end{tabular}
\end{table}

This benchmark illustrates that even in a real-life application super-linear scaling is possible to achieve for a sufficiently large number of processes.
This is due to the operations that are needed in order to perform the Wiener filtering:
    basic point-wise arithmetics that do not involve any inter-process communication and
    Fourier transformations that are handled by the high-performance library \emph{FFTW}\cite{FFTW05}.
While the problem size remains constant, the amount of available CPU cache increases with the number of processes,
    which explains the super-linear scaling,
    cf.\ section \ref{sec:strong_scaling}.

\section{Summary \& Outlook}\label{sec:summary}
We introduced \textsc{d2o}, a Python module for cluster-distributed multi-dimensional numerical arrays.
It can be understood as a layer of abstraction between abstract algorithm code and actual data-distribution logic.
We argued why we developed \textsc{d2o}\ as a package following a low-level parallelization ansatz
    and why we built it on MPI.
Compared to other packages available for data parallelization, \textsc{d2o}\ has the advantage
    of being ready for action on one as well as several hundreds of CPUs,
    of being highly portable and customizable as it is built with Python,
    that it is faster in many circumstances,
    and that it is able to treat arrays of arbitrary dimension.

For the future, we plan to cover more of \emph{numpy}'s interface such that working with \textsc{d2o}\ becomes even more convenient.
Furthermore we evaluate the option to build a \textsc{d2o}\ distributor in order to support \emph{scalapy}'s block-cyclic distribution strategy directly.
This will open up a whole new class of applications \textsc{d2o}\ then can be used for.

\textsc{d2o}\ is open source software licensed under the GNU General Public License v3 (GPL-3) and is available by \url{https://gitlab.mpcdf.mpg.de/ift/D2O}.

\begin{backmatter}

\section*{Competing interests}
  The authors declare that they have no competing interests.

\section*{Author's contributions}
TS is the principal researcher for the work proposed in this article.
His contributions include the primal idea, the implementation of presented software package, the conduction of the performance tests and the writing of the article.
MG, FB and TE helped working out the conceptual structure of the software package and drafting the manuscript.
FB also played a pivotal role for executing the performance tests.
Furthermore, TE also fulfilled the role of the principal investigator.
All authors read and approved the final manuscript.

\section*{Acknowledgments}
We want to thank Jait Dixit, Philipp Franck, Reimar Leike, Fotis Megas, Martin Reinecke and Csongor Varady for useful discussions and support.
We acknowledge the support by the DFG Cluster of Excellence "Origin and Structure of the Universe" and the Studienstiftung des deutschen Volkes.
The performance tests have been carried out on the computing facilities of the Computational Center for Particle and Astrophysics (C2PAP).
We are grateful for the support by Dr. Alexey Krukau through the Computational Center for Particle and Astrophysics (C2PAP).

\bibliographystyle{bmc-mathphys}

redpajama_set_name: RedPajamaArXiv


The Piedmont Park Conservancy, a nonprofit organization, raises and invests $3 million annually to keep Piedmont Park safe, clean and green. Becoming a Local Business Member (LBM) not only supports these efforts, but connects you to Piedmont Park and its 4 million annual visitors.
All Local Business Members are featured on the piedmontpark.org and receive a window decal to showcase your support of the Piedmont Park Conservancy. Even better, memberships are tax-deductible!
The Local Business Membership program is a great way to support Piedmont Park while promoting your local business.
One annual mention on our social media sites (over 250,000 followers!) sharing your support with a quote from your business, offering discounts to Conservancy members, etc.
Visit our Local Business Members webpage to see who is supporting the Piedmont Park Conservancy!


Glory Glimpse: Misjudging Jesus, the Nazarene – What song did God give you?
Jesus of Nazareth, the Nazarene, is our focal point today. Though the Old Testament does not mention the village of Nazareth, Matthew declares that Jesus dwelt in Nazareth "that the words of the prophets might be fulfilled." Jesus referred to Himself as 'of Nazareth, demons used the title and the cross of our Lord bore the reference. That Jesus was of Nazareth—was a Nazarene—has tremendous significance for us.
Indeed, we know great good came from this despised village! As the Crusaders of the 12th century inscribed on a church erected in Nazareth, "Here the Word was made flesh." For centuries, the prophets proclaimed the coming Messiah. Some prophesies revealed His Kingly nature—some His Suffering Servant role. Our human tendency is to hear and see that which promises pleasure: preferring power to persecution is human self-preservation. It is comfortable for us to be blind to Truth when it does not fit our plans and pre-conceived ideas. Our Jesus, the Nazarene, calls us to seek the Father's perspective and His acceptance above all else. Jesus of Nazareth reminds us that God's ways our not our ways.
Just as man erroneously judged the village of Nazareth, so, too, man mis-judges Jesus. As Isaiah prophesied of the coming Messiah, Jesus was 'despised and rejected of men'. The plans of man have no room for suffering. Yet, God chose Nazareth as the hometown of the Savior because rejection of the world is the path to acceptance of God. Embracing the Nazarene, the One the world rejects, is the only way of acceptance for each one of us. Only in Him–the Beloved–can God accept us. This world, though created, loved and redeemed by Him, received Him not. The world prefers the darkness of its own plans above the Truth and Light of the Father's plans. Love of the world cuts us off from the love of the Father. May our hearts rejoice that Jesus, our Savior, accepted the path of rejection that acceptance would be ours.
Would you like a free PDF of Names of God? Fill out the form below and you will be on your way to receiving it!


Tranquil living environments can positively affect the human brain function, according to researchers at the University of Sheffield.
The research, which was published in the journal NeuroImage, uses functional brain imaging to assess how the environment impacts upon our brain functions.
The findings demonstrated that tranquil environmental scenes containing natural features, such as the sea, cause distinct brain areas to become 'connected' with one another whilst man-made environments, such as motorways, disrupt the brain connections.
The research involved academics from the University's Academic Unit of Clinical Psychiatry, Academic Radiology and the School of Architecture, along with the School of Engineering, Design and Technology at the University of Bradford and the Institute of Medicine and Neuroscience at Jülich, Germany. The team carried out functional brain scanning at the University of Sheffield to examine brain activity when people were presented with images of tranquil beach scenes and non-tranquil motorway scenes.
They utilised the fact that waves breaking on a beach and traffic moving on a motorway produce a similar sound, perceived as a constant roar, and presented the participants with images of tranquil beach scenes and non-tranquil motorway scenes while they listened to the same sound associated with both scenes.
Using brain scanning that measures brain activity they showed that the natural, tranquil scenes caused different brain areas to become 'connected' with one another – indicating that these brain regions were working in sync. However, the non-tranquil motorway scenes disrupted connections within the brain.
Study shows tranquil scenes have positive impact on brain.


Description: Wagon trains went through camp after camp, and saw many daybreaks far from the familiar. Then went across deserts, and forded rivers. This screen saver, which was made for my wagon train series, attempts to show all these aspects of a wagon train as it traveled west.
Description: a tiger in the falling snow, waiting for the springtime that is all about him. music to fit the scene and many animations. hope you will enjoy it.
Description: Watch this screensaver and you get the impression that the clouds are moving while rain falls, thunder and lightning crash. Free to try for 3 days. Registration is on my site. Has matching theme and wallpaper.
Description: Waiting for Santa to come :o) Screensaver cont. a compl.song and many animations.Also has a theme / 3logos and hotbar.
Description: A Welsh Flag for your desktop.Music may be muted.
Description: Bridge Trolls for your desktop. One dances to the music, one scurrys across the bottom in this screensaver. Music is Walk Like A Man.
Description: Shoes! Sandals! Slides! Mules! and more! These were made for walking - after midnight or anytime. Companion TRANSPARENT ANIMATED saver for my theme Just Walking - and a bit of fun into the bargain.
Description: A walking dog with a barely there mistress, a real time clock and Walking The Dog mutable music in this saver.
Description: Matching rain effect screensaver to desktop theme, hotbar, winamp, IE toolbar and wallpaper.


Six years ago, Wellborn Forest Products faced the very real possibility of bankruptcy. Under the new leadership of Tim Wellborn, the company negotiated a loan extension and within a year paid off the bank. Since then, the company has taken a series of steps to ensure steady growth.
During a long holiday weekend in 2001, WFP, a manufacturer of kitchen cabinets, took a huge risk and completely reinvented itself, changing its focus, box construction and plant layout.
"We threw away the mold and started doing everything differently," says Wellborn.
The change caused the company to lose money in 2001. But since then, WFP of Alexander City, Ala., has experienced consistent growth, doubling its volume. "During a three-year period of tremendous transition, we were also able to grow the company 120 percent," says Blake Harmon, vice president of sales.
The new DISA Cattinair finishing system is the next stage in the company's growth and evolution. It's expected to increase production by as much as 40 percent. With this finishing system and the current plant setup, the company now believes it can reach $50 to $55 million in annual sales. The new paint room has the capacity to handle the needs of the company for the next several years, says Wellborn.
When WFP went from building primarily stock cabinets to semi-custom cabinets in 2001, Wellborn expected the changes to result in some difficulties, but he says revenue generated from product built today is far greater.
With that growth and the change to a more upscale product, the company realized it needed a better finishing system. With the old conventional paint system, everything was done by hand.
The newest bottleneck from a production and quality standpoint was the paint room, says Wellborn. Fourteen months and almost $5 million later, the system became fully operational within the company's standards.
WFP opted for a rotary spray finishing system rather than a reciprocating machine for the quality of finish it provided and for the versatility and redundancy it felt was built into the system. The computer-controlled system has rotating spray guns that spray precisely on the piece of wood being coated from all angles as a part moves on the conveyor.
The company wanted its stain, sealer and clear-coat finish stations to have the same system to provide redundancy. "Let's say our rotary stain system did go down," says Wellborn. "We're not totally dead. We can't run at full production, but since there are eight guns on the sealer rotary setting, 10 can be configured to spray stain."
WFP also built elements into the system that aren't yet in use an infrared drying machine, a buffing machine that can mimic a hand-rubbed finish and a line for glazing, WFP's niche in the market.
"It's a lot easier to design in certain things when you're building a system like this than to come back and retrofit it later," says Wellborn. His goal was to set up a line capable of using waterborne paints because the industry is moving that way, he says. The conveyor, paint, drying and dust systems, and all of the piping above was powder coated in Europe, and the lines are all stainless steel in anticipation of using waterborne products.
"We think the technology we have is on the cutting edge. It's almost taking a chance out there where we are, but it's worked," says Wellborn.
Wellborn, his wife, Pam, and James Fell, the plant manager, went to France, along with WFP's paint company, Chemcraft. The Wellborn group told Cattinair what they wanted the system to do and Chemcraft worked with the Cattinair engineers to create the best setup for the company. The system arrived in the United States in January 2004, with installation complete in April.
"Having the machine in place and running is just the start on a system like this," Wellborn says. "You have to make sure the system gives you the same color you've been providing your customer base for years."
It took six months to qualify every color of the standard, says Harmon, although most of the colors were done in two months. When a color was a little off, WFP had to go back to Chemcraft and it would reformulate the color and WFP would try it again.
Everything was just right, and then WFP changed to Kremlin spray guns because it felt the application was better. "So there we were at square one again, reformulating stains," says Wellborn.
The gun angle also had to be re-determined so it would fill in paint or stain with minimum bounce off. You have to find a happy medium that works with different door styles, says Harmon. To keep the product consistent, every station has a dedicated machine operator; others are cross-trained to step in if the need arises.
It all starts with the sanders, says Wellborn, because the fanciest paint room won't matter if the wood isn't prepared properly. Parts go through two DMC Unisand 2000 computerized sanders, a DiscMaster SlipCon finishing sander and touch-up hand sanding before they are finished.
After sanding, stain is applied. Parts then go through a flash-off oven and an infrared system, and are flipped over and sent down the return line. Parts go through the line again to stain the other side. Parts then go through a pre-cleaner, which is a brush that cleans off the parts, and a pre-heater. A clear sealer is then applied to both sides, using the same process in an adjacent line. Parts then move through the finishing sander and receive touch-up hand sanding.
When the glazing line is operational, parts will then be glazed. Finally, after another pre-cleaning, parts receive a topcoat. With the topcoat, parts also go through an ultraviolet light and a final high-intensity ultraviolet light to cure the topcoat. The topcoat and sealer react and bond together, says Harmon. At this point, parts can be stacked and handled.
The conventional paint line is still running, but it is only used for thick or oversized parts and glazes. Now that the new line is up and running, the conventional system will be updated and reduced in size. WFP will use the extra space to increase and redirect the assembly line.


text:
Transformative Constitutionalism revisited
My colleague from Stellenbosch University, Prof Sandra Liebenberg has written an excellent piece on the notion of transformative constitutionalism. I could not have said it better:
The notion of 'transformative constitutionalism' has found a deep resonance in the jurisprudence of the courts, academic literature and civil society campaigns for social justice. As our constitutional institutions are feeling the strain of recent developments, it is fitting to reflect on some of the challenges which face the realisation of this transformative vision of the Constitution. …
The first challenge concerns the increasing signs of the emergence of a narrow, patriarchal nationalist identity with its characteristic penchant for the exclusion and marginalisation of 'the other'. This was most graphically manifested in the explosion of xenophobic violence earlier this year. However, its insidious presence can also be detected in the reactions of the Labour Minister and BEE leaders to the court's ruling concerning inclusion of South Africa citizens of Chinese descent in empowerment legislation, the daily 'bureaucratic violence' dished out to refugees, asylum-seekers and other categories of non-nationals in their attempts to gain access to basic services from government departments, the endemic violence against women and AIDS activists, and the horrific conditions in which prisoners are incarcerated in many prisons in South Africa. These phenomena are the antithesis of a constitutional project which values human dignity, interdependence and a diverse society.
Secondly, the statistics continue to tell the tale of increased socio-economic disparities in wealth. The on-going systemic inequality and deep conditions of poverty afflicting a large proportion of the population risks making the constitutional commitment to social justice and an improvement in people's quality of life seem hollow.
Finally, there are the subtle undermining and the not-so-subtle attacks on the foundation of a constitutional state – the rule of law and an independent judiciary. The subtle undermining refers to the trend which has emerged of many government departments failing to respect court orders. This has a long history stretching back to the government's failure to respect orders of the courts primarily in the Eastern Cape and KwaZulu-Natal to ensure that social grants are paid timeously and are not unlawfully terminated. The courts have struggled valiantly to deal with this phenomenon through a range of mechanisms such as maintaining judicial supervision over mandatory orders against government departments, making awards of constitutional damages against the relevant departments, citing government officials for contempt of court, and even threatening to make government officials and the heads of department responsible for paying the costs of cases out of their own pockets.

redpajama_set_name: RedPajamaCommonCrawl


I rank Ted Williams as one of the three greatest offensive players in the history of MLB. It can easily be debated, as some look at several other players that may or not be considered for such a distinction. That can be another story. Williams numbers speak for himself; he hit .347 for his career and no player in the history of MLB has a higher OBP than Williams' .482. Williams played 19 MLB seasons and had a career OPS of 1.116, failing to accumulate a 1.000 OPS only once in his career. That was 1959.
As great as Williams was (which does not need to be proved), he should have gotten more credit for as great of a player he was. He was never known to embrace the media and he always had a bad relationship with the fans, particularly in Boston. With the numbers that Williams put up on a year in and year out basis, it is amazing that he won just two AL MVPs during his career. He won the awards in 1946 and 1949. He also finished 2nd in the voting 4 times: in 1941, 1942, 1947 and 1957. A case can be made that he could have (and maybe should have) won the award in those four seasons as well.
In addition to his relationship with the media, Williams quietly put up similar numbers each season. The problem with that argument is the fact that he put up numbers on a consistent basis that few were putting up at that time. This was a 6 time batting champion, who led the AL in OBP 13 times, SLG 9 times and OPS 10 times. He also led the AL in OPS+ 9 times and total bases 6 years.
The first season Williams finished 2nd in the AL MVP was 1941, after his famous .406 batting average. Nobody knew at the time that it would be the last time a MLB hitter batted over .400. That was the same season that Joe DiMaggio had his famous 56 game hitting streak. Williams, that season, had a career high 1.287 OPS as he also led the AL in RS (135), HRs (37), OBP (.553), SLG (.735) and OPS+ (235). DiMaggio had a very good season, but his numbers were not better than WIlliams of 1941. DiMaggio hit (.356), 30, 125 (125 RBIs led the AL, Williams finished with 120). DiMaggio's 1.083 OPS was great, but it was over 200 points less than what Williams finished with. DiMaggio did lead the AL in total bases with 343 and had more doubles (43) than Williams (33). However, comparing the two seasons are no contest, as Williams clearly was the most valuable player. However, the Yankees won the AL Pennant (and eventually the World Series), which in today's case, may have qualified DiMaggio over Williams.
Williams followed up his 1941 season with a year that may have been considered more dominant in 1941. Williams led the AL with a .356 AVG, 36 HR, 137 RBI, 141 RS, 145 BB, .499 OBP, .648 SLG, 1.147 OPS, 216 OPS+ and 338 TB. The 1942 AL MVP went unfairly to the Yankees Joe Gordon, who did not finish ahead of Williams in a single AL category. Gordon hit .322, 18, 103 with a .900 OPS. Of course, the Yankees won the AL Pennant again. However, it was more evident that at least some of the baseball writers voted against Williams because they did not like him.
Williams finally got his MVP in 1946, when the Red Sox won their only Pennant during his career. It was also his first year back after serving in World War II and based on his numbers, it seemed he did not miss a beat. He hit .342, 38, 123. However, he did not lead the AL in any of the three categories. He led the AL in RS (142), BB (156), OBP (.497), SLG (.667), OPS (1.166), OPS+ (215) and TB (343). He beat out Detroit RHP Hal Newhouser (26-9, 1.94, 275 Ks, 1.069 WHIP) for the award.
The following season, Williams had as good, if not a better season. His .343 AVG led the AL. In fact, he won the AL Triple Crown for the second time in his career (1942) with 32, 114. He, as usual, led the AL in runs (125), BB (162), OBP (.499), SLG (.634), OPS (1.123), OPS+ (205) and TB (335). The award once again went to DiMaggio, who had a less than dominant season. The Yankees won the AL Pennant, which clearly had to be the reason DiMaggio got the award. DiMaggio hit .315, 20, 97 with 97 RS and a .913 OPS. DiMaggio failed to lead the AL in any offensive category, and if you want to use Wins Above Replacement (not one of my favorite stats), he was worth 5 runs less than Williams. In my opinion Williams got hosed more in 1947 than he did in 1942.
Williams won the MVP again in 1949. In addition to leading the AL in games played and plate appearances (career high 730 plate appearances), he led the AL in RS (150), 2B (39), HR (43), RBI (159) and walks (162). The runs scored, HR, RBI and BB were all career highs (he also had 162 walks in 1947). He hit .343 and led the AL in OBP (.490), SLG (.650), OPS (1.141), OPS+ (191) and TB (368). The total bases were also a career high. Despite the Yankees taking the AL Pennant once again, neither Phil Rizzuto nor Joe Page had seasons that could be considered comparable enough to compete for the award. However, Williams still only received 13 of the 22 first place votes.
In 1951, Williams would have another top season. He missed some time due to injury during 1950. He hit just .318, but hit 30, 126 with 109 RS and 144 BB. He once again led the AL in OBP (.464), SLG (.556), OPS (1.019), OPS+ (191) and TB (368). He finished 13th, but the well disputed award went to Yogi Berra that season. Guys like Bobby Avila and Rizzuto finished ahead of Williams, but had no business doing so based on Williams season.
Finally, in 1957, Williams finished 2nd in the vote again. He led the AL with a .388 batting average. He hit 38 HR and led also led the AL in OBP (.526) SLG (.731), OPS (1.257) and OPS+ (235). His OPS+ was the highest he had recorded since 1941. That year, Mickey Mantle hit .365, 34, 94 and actually had a higher WAR than Williams. The only one of the four 2nd place finishes a better performance beat him. It still was a phenomenal season for the Splendid Splinter.
We all know about Williams service in WWII and The Korean War. I am sure some of the writers that held such a grudge with him were some of the BBWAA writers that did not vote for him in his first season of Hall of Fame eligibility. The same crap exists today. We need to know that the writers that are voting for awards, as well as the HOF, are being objective and fair in their evaluation. Too many make it a personal thing, with some not voting for a player out of spite. Williams numbers speak for themselves. He definitely deserved to be an MVP more than 2 times.


Every manufacturer of laminate flooring provides very specific instructions on the proper method of installing their product. Laminate flooring is designed to be installed by the DIY home handyman who is capable of completing simple home improvement projects.
There are however, some common dos and don'ts that apply to the installation of any laminate floor. Surprisingly many people ignore these instructions, believing that they know better than the manufacturers.
Laminate flooring must acclimatize to your home. This is done by placing the boxes in the room where you are going to install the laminate for a minimum period of 48 to 72 hours. Failure to do this will cause floors to buckle.
Note: In the room does not mean in the house or garage! It means, precisely what it says. The laminate must be allowed to acclimatize in the room where it is to be installed.
Additional information on acclimatizing flooring products.
There must be a gap between walls and any other obstacle, (columns, stair risers, heat vents, etc.) of between 5/16″ and 1/2″ depending on the manufacturer. This allows for the expansion of the floor. If a gap is not left, the floor will buckle.
If the laminate flooring is going to be installed on a concrete slab floor where there is earth below the concrete, a vapor barrier must be placed between the concrete and the laminate. The seams should be taped with a moisture resistant tape. Laminate flooring should not be installed on any floor that has a drain.
Some manufacturers will allow the installation of laminate flooring on carpet. The carpet should be a hard twist (Berber), or a very low pile (commercial carpet). If the underlayment is spongy, the flooring will separate when walked on.
The smallest piece of flooring should never be less than 8 inches long and the width should never be less than 2 inches. Measure your room before you start the installation as it may be necessary to cut the first board in length and the first row of boards in width to avoid having small pieces.
Joints should be staggered a minimum of 16 inches apart and a minimum of 3 rows for repeating of the joint.
If installing base board or quarter round on top of the laminate, do not nail through the laminate. The nails must be driven in a horizontal plane to the floor, not vertical. A laminate floor must be allowed to expand and contract.
Repair any and all squeaks and soft spots in the sub-floor before installing the laminate. Laminate flooring will not correct squeak or soft spot problems.
Do not continue the installation of laminate flooring through a doorway that is less than 48 inches in width, use a transition molding.
Do not hammer on the groove end on the board – ever!
Always start the installation of the laminate flooring with the groove side and end towards the walls.
Always use a tapping block to help tighten joints, but do not over hammer or you will damage the surface of the laminate at the joints.
Floors must be level, no more than 3/16" slope over a ten foot span.
Floors must be flat. Any dips in the floor should be filled in with self-leveling compound and left to dry at least 72 hours.
Read and follow the manufacturer's instructions and you will end up with a quality floor that will last for years.


Please allow us 3-5 days to process your order until it is ready for delivery. All treats are made fresh to order and perishable. We recommend storing in your fridge and devouring within 5 days.
Shipping within Los Angeles or Orange County, California? Select hand delivery at checkout and we'll make an arrangement to hand deliver it for a $5 delivery fee.


I was with my 7 year old boy on a short photography trip. He really wanted to experience what a photography trip was like so I brought him here as I knew that he would be overwhelmed by the view. I feel that the view of the Colorado River from Deadhorse Point is in many ways much more dramatic than even the mighty Grand Canyon. On this morning there were no clouds and so the best light was this gentle glow that took place about 20 minutes before the sun rose. Photo © copyright by Erik Stensland.


At PFI Western Store home of BootDaddy, you'll find the Cinch clothing and accessories you love at great prices. Cinch prides itself in producing quality western shirts and jeans that are fashionable and durable. We are proud to carry men's Cinch clothing, ladies Cruel Girl clothing, and kids Cinch clothing as well. For those that live and breathe the western lifestyle Cinch clothing is a top choice.
Shop our Cinch products online and have your order delivered right to your door, or come by our store and check out for yourself how the Cinch brand puts fit and styling first.


Bihar State Power Holding Company Ltd. (BSPHCL) has released recruitment notification for filling various posts of Junior Engineer (Electrical and Civil) for its Head Offices and Field Offices. For this recruitment process, online applications are invited from well qualified Indian candidates under certain eligibility conditions.
Pay Scale: Consolidated pay of Rs. 25,900 per month during probation and will be posted in the pay scale of Level 7 as per 7th PRC Rs. 47,600 per month after successful completion of the probation.
Essential Qualification: Applicants are required to possess full time 3 years diploma in Electrical Engineering from a Government recognized Institute / College duly recognized by State Govt. / Central Govt. approved by AICTE.
Age Limit: 18 years – 37 years as on 1 January 2018 and relaxation in upper age limit will be provided as per Government rules.
Candidates fill their online application form completely in the prescribed format, at the official website of Bihar State Power Holding Company Ltd. (BSPHCL).


Providing packaging solutions to all industries has been challenging at times. Our resilient team members have proven their ability to deliver time after time. Below are many examples of the industries we serve and the solutions we provide.


text:
Nepal Energy Forum An independent forum and an on-line channel for the Nepal energy markets(Hydropower, Wind, Biomass, Solar, Petroleum).
Chameliya charges transmission line
Kathmandu, November 23
The Chameliya hydropower project has started the operation of its 133kV transmission line from Balanch of Darchula to Attariya of Kailali district.
The project announced that it successfully brought the 131-kilometre-long transmission line into operation at 1:55pm today.
"Chameliya hydropower project transmission line was charged at 132kV this afternoon," Energy Secretary Anup Kumar Upadhyaya informed.
As per its schedule, the project has targeted to generate electricity from December 12, if everything goes as planned. According to Ajay Kumar Dahal, chief of the 30-megawatt Darchula-based project, Chameliya aimed to charge the transmission line before generating energy.
Through this transmission, the generated electricity from the project will be connected to the national grid.
The project is set to test its tunnel in the next few days, Dahal said. "After completion of first phase of construction works, we will start the test operation of the project. It will take another 15 to 20 days to complete the whole construction process," Dahal informed.
Chameliya hydropower project had suffered a long delay after its 843-metre tunnel caved in due to 'technical fault' followed by a dispute with the Chinese contractor China Gezhouba Group Corporation (CGGC). CGGC, the contractor of the Chameliya Hydropower Project, had reopened the tunnel that had caved in by mudslides in March first week. The works of reopening the 843-metre section of the tunnel that had caved in started on September 28, 2015.
The project had successfully tested its dam radial gates and intake gates in the first week of May.
The project, which was initiated in January of 2007, was supposed to be completed in May of 2011. But, it has already been six years since the initial deadline expired, which has caused revenue loss of around Rs two billion every year to Nepal Electricity Authority (NEA).
NEA has been bearing a huge loss not only in terms of revenue but also due to increase in foreign exchange rate owing to the project construction delays. Nepali currency has depreciated heavily vis-à-vis the US dollar from Rs 72 in 2007 to around Rs 104 at present. Roughly per megawatt cost of the Chameliya Hydroelectricity Project will be above Rs 500 million (including the cost of road connectivity to the project area, local electrification and transmission line), as per NEA officials. The cost is significantly higher than the normal cost of Rs 150 million per megawatt.
Source: The Himalayan Times
Previous articleबुढीगण्डकी अब स्वदेशी लगानीमा
Next articleबूढीगण्डकीका लागि सवा १३ अर्ब इन्धन कर संकलन
OnlineNEF@Nepal
Import of electric vehicles plunges heavily in first five months of current FY
खिम्ती आयोजना स्वामित्व हस्तान्तरण प्रकरण
विद्युतीय तार भूमिगत कार्य थप विस्तार
'रूपा बेगनास' पम्प स्टोरेजको सर्वेक्षण अनुमति प्राधिकरणलाई
Kulekhani III period extended by nine month
बूढीगण्डकीलाई लगानीयोग्य बनाउन सरकारको एक खर्ब अनुदान
Scrapped licenses will be reinstated by the cabinet
Cash incentive instead of tax discount for hydroprojects
अरुण तेस्रोको काम २५ प्रतिशत
Decade of development Can line ministries coordinate with each other...
Uniform PDA for all hydro, mega projects
An independent forum and an on-line channel for the Nepal energy markets(Hydropower, Wind, Biomass, Solar, Petroleum).
Contact us: info@nepalenergyforum.com
NEA Electricity tariff rates
Measurement System in Nepal
Is Nepal the second richest country in water resources?
Infrastructure19
Resource2
© NEF Online Pvt. Ltd


text:
90 Day Fiance's Angela Deem Posts 'Sexy' Video Of Michael Following Troubles During Tell-All
By Mick Joest published 27 October 20
It seemed like marriage was going alright for Angela Deem and Michael Ilesanmi when 90 Day Fiancé: Happily Ever After? filming came to an end, but that changed in the TLC tell-all. The couple had hit a major roadblock on deciding whether or not they wanted children and the discussion about the process led to a massive blow-up.
Things only got worse when Angela hinted about wanting to get gastric bypass and other plastic surgeries, which clearly upset Michael. It seemed like the classic situation of a couple in the franchise headed towards a rough patch, though perhaps that's not the case. Angela uploaded a video with Michael doing a "sexy" dance, which you can check out and jam along with below.
It appears Angela wanted to get in on the latest Tik Tok musical trend of incorporating Fleetwood Mac's "Dreams" into a video with her husband Michael. The dancing is an enhanced routine compared to what 90 Day Fiancé: Happily Ever After? saw when Angela got him his tuxedo for the wedding, and I daresay the world is better for seeing him strut his stuff once again.
I do question whether the song choice was truly because of trends or intentional, especially given the nature of Angela Deem and Michael Ilesanmi's relationship. Stevie Nicks wrote the song as an outlet when Fleetwood Mac was dealing with a lot of interpersonal struggles with relationships and she had just ended a 10-year relationship with guitarist Lindsey Buckingham.
It's no secret that Angela has given Michael several ultimatums over the years they've spent together, and even in the tell-all when they were married. Was this really a fun light-hearted video, or Angela low-key telling her Nigerian husband that he can "go his own way" if he's not on board with her ground rules for marriage?
My assumption is that Angela is much more likely to directly confront Michael than send veiled Tik Tok messages via Instagram, so I'm going to subscribe to the notion that these two are having fun and enjoying the married life despite whatever other issues and distance is between them. Maybe they even found a solution to the child problem, or that's just a battle they're saving for another day down the stretch.
Angela and Michael being on good terms is positive news for fans, as the 90 Day Fiance franchise and its cast have had a bad run as of late. Between Larissa Dos Santos Lima being booted from the show and all of Deavan Clegg and Jihoon Lee's drama, it certainly doesn't hurt to have Michael dancing on the internet to remind fans this franchise can be fun at times as well.
90 Day Fiancé: HEA Strikes Back! airs on TLC Mondays at 8:00 p.m. ET. Continue to stick with CinemaBlend for all the latest in the franchise, and for more news in television and movies.
Up next: 2020 Fall TV Premiere Schedule: List Of New And Returning Shows
Mick Joest
Mick likes good television, but also reality television. He grew up on Star Wars, DC, Marvel, and pro wrestling and loves to discuss and dissect most of it. He's been writing online for over a decade and never dreamed he'd be in the position he is today.


text:
Q: How to integrate this integral by contour integral? How to show that
\begin{equation*}
\int_{-\infty}^{\infty} \frac{x\sin x}{x^2+a^2}dx=\pi e^{-a}\text{ for all }a>0
\end{equation*}
by using contour integral?

A: You should integrate the analytic continuation:
$$f(z) = \frac{ze^{iz}}{z^2+a^2}$$
On the following contour.

There are 2 poles or order 1, but just one pole is contained in the region enclosed by $\Gamma$.
It has residue $\dfrac{1}{2e^a}$
According to the mighty residue theorem
$$\int_\Gamma \frac{ze^{iz}}{z^2+a^2}\text{d}z = 2\pi i \cdot \frac{1}{2e^a} = i \color{green}{\frac{\pi}{e^a}}$$
Then evaluate the integral over the contours:
Starting with $\Gamma_1$
$$\left|\int_{\Gamma_1} \frac{ze^{iz}}{z^2+a^2}\text{d}z \right| \leqslant \int_0^{\pi} \frac{R^2e^{-R\sin \theta}}{R^2-a^2} \text{d} \theta$$
Where you can split $\int^\pi_0 e^{-R\sin\theta} \text{d} \theta$ as:
$$\begin{align}
\int^\pi_0 e^{-R\sin\theta} \text{d} \theta &= \int_0^{\frac{\pi}{2}} e^{-R\sin\theta} \text{d} \theta + \int_{\frac{\pi}{2}}^\pi e^{-R\sin\theta} \text{d} \theta\\
& \leqslant \int_0^{\frac{\pi}{2}} e^{-R\frac{2}{\pi}{\theta}} \text{d} \theta \\
& \leqslant \frac{-\pi}{2R}(e^{-R}-1)
\end{align}$$
While $\Gamma_2$
$$\int_{-R}^{R} \frac{xe^{ix}}{x^2+a^2}\text{d}x = \int_{-R}^{R} \frac{x\cos{x}}{x^2+a^2}\text{d}x + i \cdot \color{green}{\int_{-R}^{R} \frac{x\sin x}{x^2+a^2}\text{d}x} $$
When taking the limit, the green parts will be identical and this proves the statement.
You should also prove convergence of the last integral when taking the limit to infinity

redpajama_set_name: RedPajamaStackExchange


They were actually invited to two shows, he said. At the first one, his co-worker talked while Dave, wearing his lab coat, "took some body-fat measures on her volunteer while she made jokes and her band played the David Rose 'Stripper' theme song as this old guy pulled up his flannel shirt for me to measure his stomach.
"She had an interesting sense of humor," he added.
Dave's co-worker opted out of the second show because he was sensitive to "getting too much flak as this was the calm before the storm of the mega scandal," he recalled. Instead, Dave had "two young lady work assistants who demonstrated the exercises and who took the first stress tests.