text:
\section{Estimating 2D object pose.}

\paragraph{Evaluation metrics.}
In addition to the human 3D pose, we evaluate also the accuracy of the estimated 2D object pose.
For quantitative evaluation, we calculate the 2D end-point reprojection error and proportion of detections, as well as the bounding box AP$_{50}$ (IoU threshold $0.5$). Reference 2D end-point locations were annotated manually using the VIA annotator \cite{dutta2016via} in 3 keyframes per video. The bounding boxes are defined as orthogonal \jiri{Josef: is this the correct adjective?} boxes with the end-points in opposite corners \jiri{Josef: This seems to be the most reasonable definition so that all bounding boxes are comparable}.

\paragraph{Results}

We compare the final estimate by trajectory optimization and the initial estimate by Mask R-CNN (see Table~\ref{tb:bboxerror}).
The results show that the human 3D pose contains surprisingly rich information to disambiguate the 2D pose for objects

\begin{table}[t]
\footnotesize
\centering
\caption{\small \textbf{2D object end-point localization results on the Handtool dataset.} For each tool we report the 2D end-point reprojection error (in pixels) / proportion of detections.}
\label{tb:endpointserror}
\begin{tabular}{lccccc}
\hline
\jiri{?} & Barbell & Scythe & Spade & Avg \\ \hline
Mask R-CNN & 44.3/0.77 & 24.4/0.87 & 23.9/0.83 & ... /0.82 \\
Ours & \textbf{.../1.0} & \textbf{...} & \textbf{...} & \textbf{...} \\ \hline
\end{tabular}
\end{table}

\begin{table}[t]
\footnotesize
\centering
\caption{\small \textbf{2D object end-point localization results on the Handtool dataset.} For each tool we report the bounding box AP$_{50}$. \jiri{Update Mask R-CNN numbers using just 5 videos selected by Zongmian.} \jiri{* denotes padding of 5 pixels, ** 10 pixels, *** 25 pixels}}
\label{tb:bboxerror}
\begin{tabular}{lcccccc}
\hline
AP$_{50}$ & Barbell & Hammer & Scythe & Spade & mAP$_{50}$ \\ \hline
Mask R-CNN* & $0.53$ & $0.6$ & $0.93$ & $0.8$ & $-$ \\
Ours*  & \textbf{$0.4$} & \textbf{$0.47$} & \textbf{$0.8$} & \textbf{$0.2$} & \textbf{$-$} \\ \hline
Mask R-CNN** & $0.6$ & $0.6$ & $0.93$ & $0.87$ & $-$ \\
Ours**  & \textbf{$0.47$} & \textbf{$0.47$} & \textbf{$0.8$} & \textbf{$0.27$} & \textbf{$-$} \\ \hline
Mask R-CNN** & $0.67$ & $0.6$ & $0.93$ & $0.93$ & $-$ \\
Ours**  & \textbf{$0.73$} & \textbf{$0.73$} & \textbf{$1.0$} & \textbf{$0.53$} & \textbf{$-$} \\ \hline
\end{tabular}
\end{table}

\section{Introduction}\label{sec:introduction}
People can easily learn how to break concrete with a sledgehammer or cut hay using a scythe by observing other people performing such tasks in instructional videos, for example.
They can also easily perform the same task in a different context.
This involves advanced visual intelligence capabilities such as recognizing and interpreting complex person-object interactions that achieve a specific goal.
Understanding such complex interactions is a key to building autonomous machines that learn how to interact with the physical world by observing people.

This work makes a step in this direction and describes a method to estimate the 3D motion and actuation forces of a person manipulating an object given a single unconstrained video as input, as shown in Figure~\ref{fig:teaser}.
This is an extremely challenging task.
First, there are inherent ambiguities in the 2D-to-3D mapping from a single view: multiple 3D human poses correspond to the same 2D input.
Second, human-object interactions often involve contacts, resulting in discontinuities in the motion of the object and the human body part in contact. For example, one must place a hand on the hammer handle before picking the hammer up. The contact motion strongly depends on the physical quantities such as the mass of the object and the contact forces exerted by the hand, which renders modeling of contacts a very difficult task.
Finally, the tools we consider in this work, such as hammer, scythe, or spade, are particularly difficult to recognize due to their thin structure, lack of texture, and frequent occlusions by hands and other human parts.

To address these challenges, we propose a method to jointly estimate the 3D trajectory of both the person and the object by visually recognizing contacts in the video and modeling the dynamics of the interactions.
We focus on rigid stick-like hand tools (e.g.~hammer, barbell, spade, scythe) with no articulation and approximate them as 3D line segments.
Our key idea is that, when a human joint is in contact with an object, the object can be integrated as a constraint on the movement of the human limb.
For example, the hammer in Figure~\ref{fig:teaser} provides a constraint on the relative depth between the person's two hands.
Conversely, 3D positions of the hands in contact with the hammer provide a constraint on the hammer's depth and 3D rotation.
To deal with contact forces, we integrate physics in the estimation by modeling dynamics of the person and the object.
Inspired by recent progress in humanoid locomotion research~\cite{carpentier2018multi}, we formulate person-object trajectory estimation as an optimal control problem given the contact state of each human joint.
We show that contact states can be automatically recognized from the input video using a deep neural network.

\section{Related work}
Here we review the key areas of related work in both computer vision and robotics literature.

\noindent \textbf{Single-view 3D pose estimation } aims to recover the 3D joint configuration of the person from the input image.
Recent human 3D pose estimators either attempt to build a \textit{direct mapping} from image pixels to the 3D joints of the human body or break down the task into \textit{two stages}: estimating pixel coordinates of the joints in the input image and then lifting the 2D skeleton to 3D.
Existing direct approaches either rely on generative models to search the state space for a plausible 3D skeleton that aligns with the image evidence \cite{sidenbladh2000stochastic,gammeter2008articulated,gall2010optimization} or, more recently, extract deep features from images and learn a discriminative regressor from the 2D image to the 3D pose \cite{hmrKanazawa18,moreno20173d,pavlakos2017coarse,tekin2016direct}.
Building on the recent progress in 2D human pose estimation \cite{newell2016stacked,newell2017associative,insafutdinov2016deepercut,cao2017realtime}, two-stage methods have been shown to be very effective \cite{akhter2015pose,zhou2016sparseness,bogo2016keep,chen20173d} and achieve state-of-the-art results \cite{martinez2017simple} on 3D human pose benchmarks~\cite{h36m_pami}.
To deal with depth ambiguities, these estimators rely on good pose priors, which are either hand-crafted or learnt from large-scale MoCap data~\cite{zhou2016sparseness,bogo2016keep,hmrKanazawa18}. However, unlike our work, these methods do not consider explicit models for 3D person-object interactions with contacts.

\noindent \textbf{Understanding human-object interactions } involves both recognition of actions and  modeling of interactions.
In action recognition, most existing approaches that model human-object interactions do not consider 3D, instead model interactions and contacts in the 2D image space~\cite{gupta2009observing,delaitre2011learning,yao2012recognizing,prest2013explicit}.
Recent works in scene understanding~\cite{jiang2013hallucinated,fouhey2014people} consider interactions in 3D but have focused on static scene elements rather than manipulated objects as we do in this work.
Tracking 3D poses of people interacting with the environment has been demonstrated for bipedal walking~\cite{brubaker2007physics,brubaker2009estimating} or in sports scenarios~\cite{videomocap2010}.
However, these works do not consider interactions with objects. Furthermore,~\cite{videomocap2010} requires manual annotation of the input video.

There is also related work on modeling person-object interactions in robotics~\cite{tassa2012} and computer animation~\cite{boulic1990}.
Similarly to people, humanoid robots interact with the environment by creating and breaking contacts~\cite{herdt2010}, for example, during walking.
Typically, generating artificial motion is formulated as an optimal control problem, transcribed into a high-dimensional numerical optimization problem, seeking to minimize an objective function under contact and feasibility constraints~\cite{diehl2006,schultz2010modeling}.
A known difficulty is handling the non-smoothness of the resulting optimization problem introduced by the creation and breaking of contacts~\cite{westervelt2003hybrid}.
Due to this difficulty, the sequence of contacts is often computed separately and not treated as a decision variable in the optimizer~\cite{kuffner2005motion,tonneau2018}.
Recent work has shown that it may be possible to decide both the continuous movement and the contact sequence together, either by implicitly formulating the contact constraints~\cite{posa2014direct} or by using invariances to smooth the resulting optimization problem~\cite{mordatch2012discovery,winkler2018gait}.

In this paper, we take advantage of rigid-body models introduced in robotics and formulate the problem of estimating 3D person-object interactions from monocular video as an optimal control problem under contact constraints.
We overcome the difficulty of contact irregularity by first identifying the contact states from the visual input, and then localizing the contact points in 3D via our trajectory estimator.
This allows us to treat multi-contact sequences (like walking) without manually annotating the contact phases.

\noindent \textbf{Object 3D pose estimation} methods often require depth or RGB-D data as input~\cite{tejani2014latent,doumanoglou2016,hinterstoisser2016going}, which is restrictive since depth information is not always available (e.g.~for outdoor scenes or specular objects), as is the case of our instructional videos.
Recent work has also attempted to recover object pose from RGB input only~\cite{brachmann2016uncertainty,rad2017bb8,posecnn2017,deepim2018eccv,oberweger2018eccv,grabner2018cvpr,rad2018cvpr}.
However, we found that the performance of these methods is limited for the stick-like objects we consider in this work.
Instead, we recover the 3D pose of the object via localizing and segmenting the object in 2D, and then jointly recovering the 3D trajectory of both the human limbs and the object. As a result, both the object and the human pose help each other to improve their joint 3D trajectory by leveraging the contact constraints.

\noindent \textbf{Instructional videos.}
Our work is also related to recent efforts in learning form Internet instructional videos \cite{malmaud2015s,Alayrac16unsupervised, Alayrac16unsupervised} that aim to segment input videos into clips containing consistent actions.
In contrast, we focus on extracting a detailed representation of the object manipulation in the form of a 3D person-object trajectory with contacts and underlying manipulation forces.

\begin{figure*}[t]
    \centering
    \includegraphics[width=\textwidth]{system_model_option_3_2.pdf}
    \caption{\small Overview of the proposed method. In the recognition stage, the system estimates from the input video the person's 2D joints, the hammer's 2D endpoints and the contact states of the individual joints.
    The human joints and the object endpoints are visualized as colored dots in the image.
    Human joints recognized as in contact are shown in green, joints not in contact in red.
    In the estimation stage, these image measurements are fused in a trajectory estimator to recover the human and object 3D motion together with the contact positions and forces.
    }
    \vspace*{-3mm}
    \label{fig:system_model_scheme_1}
\end{figure*}

\section{Approach overview}
We are given a video clip of a person manipulating an object or in another way interacting with the scene.
Our approach, illustrated in Figure~\ref{fig:system_model_scheme_1}, receives as input a sequence of frames and automatically outputs the 3D trajectories of the human body, the manipulated object, and the ground plane. At the same time, it localizes the contact points and recovers the contact forces that actuate the motion of the person and the object.
Our approach proceeds along two stages.
In the first, \textit{recognition stage}, we extract 2D measurements from the input video. These consist of 2D locations of human joints, 2D locations of a small number of predefined object endpoints, and contact states of selected joints over the course of the video.
In the second, \textit{estimation stage}, these image measurements are then fused in order to estimate the 3D motion, 3D contacts, and the controlling forces of both the person and the object.
The person and object trajectories, contact positions, and contact forces are constrained jointly by our carefully designed contact motion model, force model, and dynamics equations.
Finally, the reconstructed object manipulation sequence can be applied to control a humanoid robot via behavior cloning.

In the following, we start in Section~\ref{sec:main_stage} by describing the estimation stage giving details of the formulation as an optimal control problem.
Then, in Section \ref{sec:extract_2d_measurements} we give details of the recognition stage including 2D human pose estimation, contact recognition, and object 2D endpoint estimation.
Finally, we describe results in Section \ref{sec:results}.

\section{Estimating person-object trajectory under contact and dynamics constraints}\label{sec:main_stage}
We assume that we are provided with a video clip of duration $T$ depicting a human subject manipulating an object. We encode the 3D poses of the human and the object, including joint translations and rotations, in the configuration vectors $q^\mathrm{h}$ and $q^\mathrm{o}$, for the human and the object respectively.
We define a constant set of $K$ contact points between the human body and the object (or the ground plane).
Each contact point corresponds to a human joint, and is activated whenever that human joint is recognized as in contact.
At each contact point, we define a contact force $f_k$, whose value is non-zero whenever the contact point $k$ is active.
The state of the complete dynamical system is then obtained by concatenating the human and the object joint configurations $q$ and velocities $\dot{q}$ as
$x \coloneqq \left(q^\mathrm{h}, q^\mathrm{o}, \dot{q}^\mathrm{h}, \dot{q}^\mathrm{o}\right)$.
Let $\tau^\mathrm{h}_\mathrm{m}$ be the joint torque vector describing the actuation  by human muscles. This is a $n_q-6$ dimensional vector where $n_q$ is the dimension of the human body configuration vector. We define the control variable $u$ as the combination of the joint torque vector together with contact forces at the $K$ contact joints, $u \coloneqq \left(\tau^\mathrm{h}_\mathrm{m}, f_k, k=1,...,K\right)$.
To deal with sliding contacts, we further define a contact state $c$ that consists of the relative positions of all the contact points with respect to the object (or ground) in the 3D space.

Our goal is two-fold. We wish to (i) estimate smooth and consistent human-object and contact trajectories $\underline{x}$ and $\underline{c}$, while (ii) recovering the control $\underline{u}$ which gives rise to the observed motion\footnote{In this paper, trajectories are denoted as underlined variables, e.g. $\underline{x},\underline{u}~\text{or}~\underline{c}$.}.
This is achieved by jointly optimizing the 3D trajectory $\underline{x}$, contacts $\underline{c}$, and control $\underline{u}$ given the measurements (2D positions of human joints and object endpoints together with contact states of human joints) obtained from the input video.
The intuition is that the human and the object's 3D poses should match their respective projections in the image while their 3D motion is linked together by the recognized contact points and the corresponding contact forces.
In detail, we formulate person-object interaction estimation as an optimal control problem with contact and dynamics constraints:
\begin{align}
    \underset{\underline{x},\underline{u},\underline{c}}{\text{minimize}} &\quad \sum_{e \in \{\mathrm{h}, \mathrm{o}\}}{\int_{0}^{T}{l^{e}\left(x, u, c\right)\mathrm{d}t}}, \label{eq:general_problem}\\
    \text{subject to} &\quad  \kappa(x, c) = 0 \quad \text{(contact motion model)}, \label{eq:contact_motion_model}\\
    & \quad \dot{x} = f\left(x, c, u\right)  \quad \text{(full-body dynamics)}, \label{eq:full_body_dynamics}\\
    &\quad u \in \mathcal{U}\quad \text{(force model)},\label{eq:force_model}
\end{align}
where $e$ denotes either `$\mathrm{h}$' (human) or `$\mathrm{o}$' (object), and the constraints \eqref{eq:contact_motion_model}-\eqref{eq:force_model} must hold for all $t\in[0,T]$.
The loss function $l^e$ is a weighted sum of multiple costs capturing (i) the data term measuring the discrepancy between the observed and re-projected 2D joint and object endpoint positions, (ii) the prior on the estimated 3D poses, (iii) the physical plausibility of the motion and (iv) the smoothness of the trajectory.
Next, we in turn describe these cost terms as well as the insights leading to their design choices.
For simplicity, we ignore the superscript $e$ when introducing a cost term that exists for both the human $l^\mathrm{h}$ and the object $l^\mathrm{o}$ component of the loss.
We describe the individual terms using continuous time notation as used in the overall problem formulation \eqref{eq:general_problem}.
A discrete version of the problem as well as the optimization and implementation details are relegated to Section \ref{sec:optimization}.

\subsection{Data term: 2D re-projection error}\label{sec:likelihood_of_the_3d_pose}
We wish to minimize the re-projection error of the estimated 3D human joints and 3D object endpoints with respect to the 2D measurements obtained in each video frame.
In detail, let $j=1,...,N$ be human joints or object endpoints and $p^{\mathrm{2D}}_j$ their 2D position observed in the image.
We aim to minimize the following \textit{data term}
\begin{align}
    l_\mathrm{data} =  \sum_j{\rho\left(p^{\mathrm{2D}}_{j} - P_\mathrm{cam}(p_{j}(q))\right)} \label{eq:data},
\end{align}
where $P_\mathrm{cam}$ is the camera projection matrix and $p_j$ the 3D position of joint or object endpoint $j$ induced by the  person-object configuration vector $q$.
To deal with outliers, we use the robust Huber loss, denoted by $\rho$.

\subsection{Prior on 3D human poses}
\label{sec:pose_prior}
A single 2D skeleton can be a projection of multiple 3D poses, many of which are unnatural or impossible exceeding the human joint limits.
To resolve this, we incorporate into the human loss function $l^\mathrm{h}$ a pose prior similar to~\cite{bogo2016keep}.
The pose prior is obtained by fitting the SMPL human model \cite{loper2015smpl} to the CMU MoCap data \cite{CmuMarkerDataset} using MoSh \cite{Loper:SIGASIA:2014} and fitting a Gaussian Mixture Model (GMM) to the resulting SMPL 3D poses.
We map our human configuration vector $q^\mathrm{h}$ to a SMPL pose vector $\theta$ and compute the likelihood under the pre-trained GMM
\begin{align}
l^\mathrm{h}_\mathrm{pose} &= -\log\left(p(q^\mathrm{h}; \text{GMM})\right).\label{eq:human_pose}
\end{align}
During optimization, $l^\mathrm{h}_\mathrm{pose}$ is minimized in order to favor more plausible human poses against rare or impossible ones.

\subsection{Physical plausibility of the motion}\label{sec:physical_plausibility}
Human-object interactions involve contacts coupled with interaction forces, which are not included in the data-driven cost terms \eqref{eq:data} and \eqref{eq:human_pose}.
Modeling contacts and physics is thus important to reconstruct object manipulation actions from the input video.
Next, we outline models for describing the motion of the contacts and the forces at the contact points. Finally, the contact motions and forces, together with the system state $\underline{x}$, are linked by the laws of mechanics via the dynamics equations, which constrain the estimated person-object interaction. This full body dynamics constraint is detailed at the end of this subsection.

\paragraph{Contact motions.}
In the recognition stage, our contact recognizer predicts, given a human joint (for example, left hand, denoted by $j$), a sequence of contact states $\delta_j: t \longrightarrow \{1,0\}$.
Similarly to \cite{carpentier2018multi}, we call a \textit{contact phase} any time segment in which $j$ is in contact, i.e., $\delta_j=1$.
Our key idea is that the 3D distance between human joint $j$ and the active contact point on the object (denoted by $k$) should remain zero during a contact phase:
\begin{align}
     \left\|p^\mathrm{h}_j(q^\mathrm{h}) - p^\mathrm{c}_k(x, c)\right\|=0\quad  \text{(point contact)},\label{eq:point_contact}
\end{align}
where $p^\mathrm{h}_{j}$ and $p^\mathrm{c}_k$ are the 3D positions of joint $j$ and object contact point $k$, respectively. Note that position of the object contact point $p^\mathrm{c}_k(x, c)$ depends on the state vector $x$ describing the human-object configuration and the relative position $c$ of the contact along the object.
The position of contact $p^\mathrm{c}_k$ is subject to a feasible range denoted by $\mathcal{C}$.
For stick-like objects such as hammer, $\mathcal{C}$ is approximately the 3D line segment representing the handle. For the ground, the feasible range $\mathcal{C}$ is a 3D plane.
In practice, we implement $p^\mathrm{c}_k\in \mathcal{C}$ by putting a constraint on the trajectory of relative contact positions $\underline{c}$.

Equation \eqref{eq:point_contact} applies to most common cases where the contact area can be modeled as a point.
Examples include the hand-handle contact and the knee-ground contact.
To model the \textit{planar contact} between the human sole and ground, we approximate each sole surface as a planar polygon with four vertices, and apply the point contact model at each vertex.
In our human model, each sole is attached to its parent ankle joint, and therefore the four vertex contact points of the sole are active when $\delta_\mathrm{ankle}=1$.

The resulting overall contact motion function $\kappa$ in problem \eqref{eq:general_problem} is obtained by unifying the point and the planar contact models:
\begin{align}
\kappa(x, c)=\sum_{j}\sum_{k\in \phi(j)}\delta_j\left\|T^{(kj)} \left(p^\mathrm{h}_j(q^\mathrm{h})\right) - p^\mathrm{c}_k(x, c)\right\|, \label{eq:contact_model}
\end{align}
where the external sum is over all human joints.
The internal sum is over the set of active object contact points mapped to their corresponding human joint $j$ by mapping $\phi(j)$.
The mapping $T^{(kj)}$ translates the position of an ankle joint $j$ to its corresponding $k$-th sole vertex; it is an identity mapping for non-ankle joints.

\paragraph{Contact forces.}
During a contact phase of the human joint $j$, the environment exerts a contact force $f_k$ on each of the active contact points in $\phi(j)$.
$f_k$ is always expressed in contact point $k$'s local coordinate frame.
We distinguish two types of contact forces: (i) 6D spatial forces exerted by objects and (ii) 3D linear forces due to ground friction.
In the case of object contact, $f_k$ is an unconstrained 6D spatial force with 3D linear force and 3D moment.
In the case of ground friction, $f_k$ is constrained to lie inside a 3D friction cone $\mathcal{K}^3$ (also known as the quadratic Lorentz ``ice-cream'' cone \cite{carpentier2018multi}) characterized by a positive friction coefficient $\mu$.
In practice, we approximate $\mathcal{K}^3$ by a 3D pyramid spanned by a basis of $N=4$ generators, which allows us to represent $f_k$ as the convex combination $f_k = \sum_{n=1}^{N}{\lambda_{kn}g^{(3)}_n}$, where $\lambda_{kn}\geq 0$ and $g^{(3)}_n$ with $n=1,2,3,4$ are the 3D generators of the contact force.
We sum the contact forces induced by the four sole-ground contact points and express a unified contact force in the ankle's frame:
\begin{align}
    f_j=\sum_{k=1}^4
    \begin{pmatrix}
        f_k \\
        p_k\times f_k
    \end{pmatrix}
    =\sum_{k=1}^{4}\sum_{n=1}^{N}\lambda_{jkn}g^{(6)}_{kn},
\end{align}
where $p_k$ is the position of contact point $k$ expressed in joint $j$'s (left/right ankle) frame, $\times$ is the cross product operator, $\lambda_{jkn}\geq 0$, and $g^{(6)}_{kn}$ are the 6D generators of $f_j$.
Please see Appendix \ref{appendix:ground_force_generators} for additional details including the expressions of $g^{(3)}_{n}$ and $g^{(6)}_{kn}$.

\paragraph{Full body dynamics.}
The full-body movement of the person and the manipulated object is described by the Lagrange dynamics equation:
\begin{align}
    M(q)\ddot{q} + b(q, \dot{q}) = g(q) + \tau, \label{eq:dynamics_equation}
\end{align}
where $M$ is the generalized mass matrix, $b$ covers the centrifugal and Coriolis effects, $g$ is the generalized gravity vector and $\tau$ represents the joint torque contributions.
$\dot{q}$ and $\ddot{q}$ are the joint velocities and joint accelerations, respectively.
Note that \eqref{eq:dynamics_equation} is a unified equation which applies to both human and object dynamics, hence we drop the superscript $e$ here. Only the expression of the joint torque $\tau$ differs between the human and the object and we give the two expressions next.

For human, it is the sum of two contributions: the first one corresponds to the internal joint torques (exerted by the muscles for instance) and the second one comes from the contact forces:
\begin{align}
    \tau^\mathrm{h} =
    \begin{pmatrix}
        \mathbf{0}_6 \\
        \tau^\mathrm{h}_\mathrm{m}
    \end{pmatrix}
    + \sum_{k=1}^K \left(J^\mathrm{h}_k\right)^Tf_k, \label{eq:human_torque}
\end{align}
where $\tau^\mathrm{h}_\mathrm{m}$ is the human joint torque exerted by muscles, $f_k$ is the contact force at contact point $k$ and $J^\mathrm{h}_k$ is the Jacobian mapping human joint velocities $\dot{q}^\mathrm{h}$ to the Cartesian velocity of contact point $k$ expressed in $k$'s local frame.
Let $n^\mathrm{h}_q$ denote the dimension of $q^\mathrm{h}$, $\dot{q}^\mathrm{h}$ and $\ddot{q}^\mathrm{h}$, then $\tau^\mathrm{h}_\mathrm{m}$ and $J^\mathrm{h}_k$ are of dimension $n_q^h-6$ and $3\times n_q^h$, respectively.
We model the human body and the object as free-floating base systems.
In the case of human body, the six first entries in the configuration vector $q$ correspond to the 6D pose of the free-floating base (translation + orientation), which is not actuated by any internal actuators such as human muscles.
This constraint is taken into consideration by adding the zeros in Eq. \eqref{eq:human_torque}.

In the case of the manipulated object, there is no actuation other than the contact forces exerted by the human.
Therefore, the object torque is expressed as
\begin{align}
    \tau^\mathrm{o} =
    -\sum_{\text{object contact }k} \left(J^\mathrm{o}_k\right)^Tf_k, \label{eq:object_torque}
\end{align}
where the sum is over the object contact points, $f_k$ is the contact force, and $J^\mathrm{o}_k$ denotes the object Jacobian, which maps from the object joint velocities $\dot{q}^\mathrm{o}$ to the Cartesian velocity of the object contact point $k$ expressed in $k$'s local frame.
$J^\mathrm{o}_k$ is a $3\times n^\mathrm{o}_q$ matrix where $n^\mathrm{o}_q$ is the dimension of object configuration vectors $q^\mathrm{o}$, $\dot{q}^\mathrm{o}$ and $\ddot{q}^\mathrm{o}$.

We concatenate the dynamics equations of both human and object to form the overall dynamics in Eq. \eqref{eq:full_body_dynamics} in problem \eqref{eq:general_problem}, and include a \textit{muscle torque} term
$l^\mathrm{h}_\mathrm{torque} = \|\tau^\mathrm{h}_\mathrm{m}\|^2$ in the overall cost. Minimizing the muscle torque acts as a regularization over the energy consumption of the human body.

\subsection{Enforcing the trajectory smoothness}\label{sec:smoothness}
\paragraph{Regularizing human and object motion.}
Taking advantage of the temporal continuity of video, we minimize the sum of squared 3D joint velocities and accelerations to improve the smoothness of the person and object motion and to remove incorrect 2D poses.
We include the following \textit{motion smoothing} term to the human and object loss in \eqref{eq:general_problem}:
\begin{align}
  l_\mathrm{smooth} = \sum_{j}{\left(\left\|\nu_j(q, \dot{q})\right\|^2 + \left\|\alpha_j(q, \dot{q}, \ddot{q})\right\|^2\right)},
  \label{eq:cost_smooth}
\end{align}
where $\nu_j$ and $\alpha_j$ are the spatial velocity and the spatial acceleration\footnote{Spatial velocities (accelerations) are minimal and unified representations of linear and angular velocities (accelerations) of a rigid body \cite{featherstone2014rigid}. They are of dimension 6.} of joint $j$, respectively.
In the case of object, $j$ represents an endpoint on the object.
By minimizing $l_\mathrm{smooth}$, both the linear and angular movements of each joint/endpoint are smoothed simultaneously.

\paragraph{Regularizing contact motion and forces.}
In addition to regularizing the motion of the joints, we also regularize the contact states and control by minimizing the velocity of the contact points and the temporal variation of the contact force.
This is implemented by including the following \textit{contact smoothing} term in the cost function in problem \eqref{eq:general_problem}:
\begin{align}
    l^\mathrm{c}_\mathrm{smooth} =
    \sum_{j}\sum_{k\in \phi(j)}\delta_j\left(\omega_k\|\dot{c}_k\|^2+\gamma_k\|\dot{f}_k\|^2\right)\mathrm{d}t, \label{eq:contact_smooth}
\end{align}
where $\dot{c}_k$ and $\dot{f}_k$ represent respectively the temporal variation of the position and the contact force at contact point $k$.
$\omega_k$ and $\gamma_k$ are scalar weights of the regularization terms $\dot{c}_k$ and $\dot{f}_k$.
Note that some contact points, for example the four contact points of the human sole during the sole-ground contact, should remain fixed with respect to the object or the ground during the contact phase.
To tackle this, we adjust $\omega_k$ to prevent contact point $k$ form sliding while being in contact.

\subsection{Optimization}\label{sec:optimization}

\noindent \textbf{Conversion to a numerical optimization problem.}
We convert the continuous problem~\eqref{eq:general_problem} into a discrete nonlinear optimization problem using the collocation approach~\cite{biegler2010nonlinear_chap10}.
All trajectories are discretized and constraints~\eqref{eq:contact_motion_model}, \eqref{eq:full_body_dynamics}, \eqref{eq:force_model} are only enforced on the ``collocation'' nodes of a time grid matching the discrete sequence of video frames.
 The optimization variables are the sequence of human and object poses $[ x_0 ... x_T ]$, torque and force controls $[ u_1 ... u_T ]$, contact locations $[c_0 ... c_T]$, and the scene parameters (ground plane and camera matrix).
The resulting problem is nonlinear, constrained and sparse (due to the sequential structure of trajectory optimization).
We rely on the Ceres solver~\cite{ceres-solver}, which is dedicated to solving sparse estimation problems (e.g.~bundle adjustment~\cite{triggs1999bundle}), and on the Pinocchio software~\cite{carpentier2019pinocchio,pinocchioweb} for the efficient computation of kinematic and dynamic quantities and their derivatives~\cite{carpentier2018analytical}.
Additional details are given in Appendix \ref{appendix:optimization}.

\vspace*{-4mm}
\paragraph{Initialization.}
Correctly initializing the solver is key to escape from poor local minima.
We warm-start the optimization by inferring the initial configuration vector $q_k$ at each frame using the human body estimator HMR~\cite{hmrKanazawa18} that estimates the 3D joint angles from a single RGB image.

\section{Extracting 2D measurements from video}\label{sec:extract_2d_measurements}
In this section, we describe how 2D measurements are extracted from the input video frames during the first, recognition stage of our system.
In particular, we extract the 2D human joint positions, the 2D object endpoint positions and the contact states of human joints.

\vspace*{-4mm}
\paragraph{Estimating 2D positions of human joints.}\label{sec:human_2d_pose}
We use the state-of-the-art Openpose~\cite{cao2017realtime} human 2D pose estimator, which achieved excellent performance on the MPII Multi-Person benchmark~\cite{andriluka20142d}.
Taking a pretrained Openpose model, we do a forward pass on the input video in a frame-by-frame manner to obtain an estimate of the 2D trajectory of human joints, $p^\mathrm{h,2D}_j$.

\vspace*{-4mm}
\paragraph{Recognizing contacts.}\label{sec:contact_recognition}
We wish to recognize and localize contact points between the person and the manipulated object or the ground.
This is a challenging task due to the large appearance variation of the contact events in the video.
However, we demonstrate here that a good performance can be achieved  by training a contact recognition CNN module from manually annotated contact data that combine both still images and videos harvested from the Internet.
In detail, the contact recognizer operates on the 2D human joints predicted by Openpose.
Given 2D joints at video frame $i$, we crop fixed-size image patches around a set of joints of interest, which may be in contact with an object or ground.
Based on the type of human joint, we feed each image patch to the corresponding CNN to predict whether the joint appearing in the patch is in contact or not.
The output of the contact recognizer is a sequence $\delta_{ji}$ encoding the contact states of human joint $j$ at video frame $i$, i.e.\  $\delta_{ji}=1$ if joint $j$ is in contact at frame $i$ and zero otherwise.
Note that $\delta_{ji}$ is the discretized version of the contact state trajectory $\delta_j$  presented in Sec.~\ref{sec:main_stage}.

Our contact recognition CNNs are built by replacing the last layer of an ImageNet pre-trained Resnet model \cite{he2016deep} with a fully connected layer that has a binary output.
We have trained separate models for five types of joints: hands, knees, foot soles, toes, and neck.
To construct the training data, we collect still images of people manipulating tools using Google image search.
We also collect short video clips of people manipulating tools from Youtube in order to also have non-contact examples.
We run Openpose pose estimator on this data, crop patches around the 2D joints, and annotate the resulting dataset with contact states.

\vspace*{-4mm}
\paragraph{Estimating 2D object pose.}\label{sec:object_2d_endpoints}
The objective is to estimate the 2D position of the manipulated object in each video frame.
To achieve this, we build on instance segmentation obtained by Mask R-CNN~\cite{MaskRCNN}.
We train the network on shapes of object models from different viewpoints and
 apply the trained network on the test videos. The output masks and bounding boxes are used to estimate object endpoints in each frame. The resulting 2D endpoints are used as input to the trajectory optimizer. Details are given next.

In the case of barbell, hammer and scythe, we created a single 3D model for each tool, roughly approximating the shapes of the instances in the videos, and rendered it from multiple viewpoints using a perspective camera.
For spade, we annotated 2D masks of various instances of the tool in thirteen different still images.
The shapes of the rendered 3D models or 2D masks are used to train Mask R-CNN for instance segmentation of each tool.
The training set is augmented by 2D geometric transformations (translation, rotation, scale) to handle the changes in shapes of tool instances in the videos. In addition, domain randomization \cite{loing2018,tobin2017corr} is applied to handle the variance of instances and changes in appearance in the videos caused by illumination: the geometrically transformed shape is filled with pixels from a random image (foreground) and pasted on another random image (background). We utilized random images from the MS COCO dataset \cite{COCO} for this purpose.
We use a Mask R-CNN (implementation \cite{matterport_maskrcnn_2017}) model pre-trained on the MS COCO dataset and re-train the head layers for each tool.

At test time, masks and bounding boxes obtained by the re-trained Mask R-CNN are used to estimate the coordinates of tool endpoints.
Proximity to coordinates of estimated wrist joints
is used to select the mask and bounding box in case multiple candidates are available in the frame.
To estimate the main axis of the object, a line is fitted through the output binary mask.
The endpoints are calculated as the intersection of the fitted line and boundaries of the bounding box.
Using the combination of the output mask and the bounding box compensates for errors in the segmentation mask caused by occlusions.
The relative orientation of the tool (i.e.~the head vs.~the handle of the tool) is determined by spatial location of endpoints in the video frames as well as by their proximity to the estimated wrist joints.

\section{Experiments}\label{sec:results}
In this section we present quantitative and qualitative evaluation of the reconstructed 3D person-object interactions.
Since we recover not only human poses but also object poses and contact forces, evaluating our results is difficult due to the lack of ground truth forces and 3D object poses in standard 3D pose benchmarks such as \cite{h36m_pami}.
Consequently, we evaluate our motion and force estimation quantitatively on a recent Biomechanics video/MoCap dataset capturing challenging dynamic parkour motions~\cite{maldonado}.
In addition, we report joint errors and show qualitative results on our newly collected dataset of videos depicting handtool manipulation actions.

\subsection{Parkour dataset}\label{sec:parkour_dataset}
This dataset contains videos capturing human subjects performing four typical parkour actions: two-hand jump, moving-up, pull-up and a single-hand hop.
These are highly dynamic motions with rich contact interactions with the environment. The ground truth 3D motion and contact forces are captured with a Vicon motion capture system and several force plates.
The 3D motion and forces are reconstructed with frame rates of $400$Hz and $2200$Hz, respectively, whereas the RGB videos are captured in a relatively lower rate of $25$Hz, making this dataset a challenge for computer vision algorithms due to motion blur.

\vspace*{-4mm}
\paragraph{Evaluation set-up.}
We evaluate both the estimated human 3D motion and the contact forces.
For evaluating the accuracy of the recovered 3D human poses, we follow the common approach of computing the mean per joint position error (MPJPE) of the estimated 3D pose with respect to the ground truth after rigid alignment~\cite{gower1975generalized}.
We evaluate contact forces without any alignment: we express both the estimated and the ground truth 6D forces at the position of the contact aligned with the world coordinate frame provided in the dataset.
We split the 6D forces into linear and moment components and report the average Euclidean distance of the linear force and the moment with respect to the ground truth.

\begin{table}[t]
\small
\centering
\resizebox{\columnwidth}{!}{
\begin{tabular}{lccccc}
\hline
Method                     &   Jump &  Move-up &  Pull-up  &     Hop &    Avg \\ \hline
SMPLify~\cite{bogo2016keep}& 121.75 &   147.41 &   120.48  &  169.36 &  139.69 \\
HMR~\cite{hmrKanazawa18}&    111.36 &   140.16 &   132.44  &  149.64 &  135.65 \\
Ours               & \textbf{98.42} & \textbf{125.21} & \textbf{119.92}& \textbf{138.45}    & \textbf{122.11} \\ \hline
\end{tabular}}
\caption{\small Mean per joint position error (in mm) of the recovered 3D motion for each action on the Parkour dataset.}
\label{tb:mpjpe-galo-per-action}
\vspace*{-1mm}
\end{table}

\begin{table}[t]
\begin{small}
\centering
\resizebox{0.89\columnwidth}{!}{
\begin{tabular}{lcccc}
\hline
                   & L. Sole  & R. Sole         & L. Hand        & R. Hand        \\ \hline
Force (N)     &  144.23        & 138.21         &  107.91        &  113.42   \\
Moment (N$\cdot$m)          & 23.71     & 22.32           &     131.13         & 134.21   \\ \hline
\end{tabular}}
\caption{\small Estimation errors of the contact forces exerted on soles and hands on the Parkour dataset.}
\vspace*{-4mm}
\label{tb:force-errors-galo}
\end{small}
\end{table}

\paragraph{Results.}
We report joint errors for different actions in Table \ref{tb:mpjpe-galo-per-action} and compare the results with the HMR 3D human pose estimator~\cite{hmrKanazawa18}, which is used to warm-start our method.
To make it a fair comparison, we use the same Openpose 2D joints as input.
In addition, we evaluate the recent SMPLify \cite{bogo2016keep} 3D pose estimation method.
Our method outperforms both baselines by more than 10mm on average on this challenging data.
\begin{figure*}[t]
    \centering
    \includegraphics[width=\textwidth]{qualitative.jpg}
    \caption{\small Example qualitative results on the Handtool dataset. Each example shows the input frame (left) and two different views of the output 3D pose of the person and the object (middle, right).
    The yellow and the white arrows in the output show the contact forces and moments, respectively. Note how the proposed approach recovers from these challenging unconstrained videos the 3D configuration of the person-object interaction together with the contact forces and moments. {\bf For additional video results please see the project webpage~\cite{project-page}.}
    }
    \label{fig:qualitative}
\end{figure*}
Finally, Table \ref{tb:force-errors-galo} summarizes the force estimation results.
To estimate the forces we assume a generic human physical model of mass $74.6$\,kg for all the subjects.
Despite the systematic error due to the generic human mass assumption, the results in Table \ref{tb:force-errors-galo} validate the quality of our force estimation at the soles and the hands during walking and jumping.
We observe higher errors of the estimated moments at hands, which we believe is due to the challenging nature of the Parkour sequences where the entire person's body is often supported by hands.
In this case, the hand may exert significant force and torque to support the body, and a minor shift in the force direction may lead to significant errors.

\subsection{Handtool dataset}\label{sec:handtooldataset}
In addition to the Parkour data captured in a controlled set-up, we would like to demonstrate generalization of our approach to the ``in the wild'' Internet instructional videos.
For this purpose, we have collected a new dataset of object manipulation videos, which we refer to as the Handtool dataset.
The dataset contains videos of people manipulating four types of tools: {\em barbell}, {\em hammer}, {\em scythe}, and {\em spade}.
For each type of tool, we chose among the top videos returned by YouTube five videos covering a range of actions.
We then cropped short clips from each video showing the whole human body and the tool.

\vspace*{-2mm}
\paragraph{Evaluation of 3D human poses.}
For each video in the Handtool dataset, we have annotated the 3D positions of the person's left and right shoulders, elbows, wrist, hips, knees, and ankles, for the first, the middle, and the last frame.
We evaluate the accuracy of the recovered 3D human poses by computing their MPJPE after rigid alignment.
Quantitative evaluation of the recovered 3D poses is shown table \ref{tb:mpjpe-handtools}.
On average, our method outperforms the strong HMR~\cite{hmrKanazawa18} and SMPLify~\cite{bogo2016keep} baselines. However, the differences between the methods are reaching the limits of the accuracy of the manually provided 3D human pose annotations on this dataset. Videos available at project website~\cite{project-page} demonstrate that our model produces smooth 3D motion, respecting person-object contacts and capturing the interaction of the person with the tool. This is not the case for the HMR~\cite{hmrKanazawa18} and SMPLify~\cite{bogo2016keep} baselines that are applied to individual frames and do not model the interaction between the person and the tool. Example results for our method are shown in Figure~\ref{fig:qualitative}.
For additional results including examples of the main failure modes please see Appendix \ref{appendix:qualitive_results}.

\vspace*{-2mm}
\paragraph{Evaluation of 2D object poses.}
To evaluate the quality of estimated object poses, we manually annotated 2D object endpoints in every 5th frame of each video in the Handtool dataset and calculated the 2D Euclidean distance (in pixels) between each manually annotated endpoint and its estimated 2D location provided by our method.
The 2D location is obtained by projecting the estimated 3D tool position back to the image plane.
We compare our results to the output of the Mask R-CNN instance segmentation baseline \cite{MaskRCNN} (which provides initialization for our person-object interaction model).
In Table~\ref{table:evaluation2dposeall2550100} we report for both methods the percentage of endpoints for which the estimated endpoint location lies within 25, 50, and 100 pixels from the annotated ground truth endpoint location.
The results demonstrate that our approach provides more accurate and stable object endpoint locations compared to the Mask R-CNN baseline thanks to modeling the interaction between the object and the person.

\begin{table}[t]
\footnotesize
\centering
\resizebox{\columnwidth}{!}{
\begin{tabular}{lcccccc}
\hline
Method                       & Barbell        & Spade         & Hammer         & Scythe         & Avg            \\ \hline
SMPLify~\cite{bogo2016keep}  & 130.69         & 135.03        & \textbf{93.43}          & \textbf{112.93}         & 118.02         \\
HMR~\cite{hmrKanazawa18}     & 105.04         & 97.18         & 96.34          & 115.42         & 103.49         \\
Ours               & \textbf{104.23} & \textbf{95.21}&        95.87 & 114.22 & \textbf{102.38} \\ \hline
\end{tabular}}
\caption{\small Mean per joint position error (in mm) of the recovered 3D human poses for each tool type on the Handtool dataset.}
\label{tb:mpjpe-handtools}
\end{table}

\begin{table}[t]
\small
\centering
\resizebox{\columnwidth}{!}{
\begin{tabular}{lcccc}
\hline
Method                     & Barbell         & Spade        & Hammer          & Scythe       \\ \hline
Mask R-CNN \cite{MaskRCNN} & 33/42/54       & 54/79/93     & 35/44/45    & 63/72/76    \\
Ours                       & \textbf{38/71/98}   & \textbf{57/86/99}   & \textbf{61/91/99}    & \textbf{69/88/98} \\ \hline
\end{tabular}}
\caption{\small The percentage of endpoints for which the estimated 2D location lies within 25/50/100 pixels (in 600$\times$400 pixel image) from the manually annotated ground truth location. }
\vspace*{-4mm}
\label{table:evaluation2dposeall2550100}
\end{table}

\vspace*{-1mm}
\section{Conclusion}
\vspace*{-2mm}
We have developed a visual recognition system that takes as input video frames together with a simple object model, and outputs a 3D motion of the person and the object including contact forces and torques actuated by the human limbs.

We have validated our approach on a recent MoCap dataset with ground truth contact forces.
Finally, we have collected a new dataset of unconstrained instructional videos depicting people manipulating different objects and have demonstrated benefits of our approach on this data.
Our work opens up the possibility of large-scale learning of human-object interactions from Internet instructional videos~\cite{Alayrac16unsupervised}.

\paragraph{Acknowledgments.}
We warmly thank Bruno Watier (Universit\'e Paul Sabatier and LAAS-CNRS) and Galo Maldonado (ENSAM ParisTech) for setting up the Parkour dataset.
This work was partly supported by the ERC grant
LEAP (No.\,336845), the H2020 Memmo project, CIFAR Learning in Machines\&Brains program, and the European Regional Development Fund under the project IMPACT (reg. no. CZ.02.1.01/0.0/0.0/15\_003/0000468).

{\small
\bibliographystyle{ieee}

redpajama_set_name: RedPajamaArXiv


text:
Twenty One Pilots Deep Dive Into 'Jumpsuit' in Behind-the-Scenes Video: Watch
Twenty One Pilots released an 8-minute behind-the-scenes making-of mini-doc on Thursday (Sept. 6) that helps pull back the curtain on the first clip from the band's upcoming 'Trench' album.
By Gil Kaufman
Gil Kaufman
More Stories by Gil Kaufman
Kelly Clarkson Goes 'Crazy' For Aerosmith on Twangy 'Kellyoke' Cover
Jamie Lynn Spears Says She 'Felt Like an Afterthought' Growing Up in Britney Spears' Shadow
'West Side Story' Actress Rachel Zegler Apologizes For Dramatic Re-Enactment of Britney Spears Tweets
twenty one pilots: Jumpsuit (Beyond the Video) Courtesy Photo
In a deep-dive look at the making of their "Jumpsuit" video, Twenty One Pilots released an 8-minute behind-the-scenes making-of mini-doc on Thursday (Sept. 6) that helps pull back the curtain on the first clip from the band's upcoming Trench album. Filled with storyboards and long, picturesque looks at the remote town in Iceland (Þórsmörk) where it was shot, the short film begins with director and frequent collaborator Andrew Donoho explaining how a 15-day trip to Iceland in 2016 helped inspire the austere look in "Jumpsuit."
"I was always shocked at the craziness and the expanse of the canyons," he says over black-and-white footage of fog-shrouded, ice-covered mountains. "You'll see it on camera, but in person it's even wilder because it makes it feel like you're totally surrounded and enveloped." When Donoho got wind of the world that TØP singer Tyler Joseph and the team had created for this album cycle he was excited because it instantly reminded him of what he'd seen on that previous trip.
Twenty One Pilots Get Funky on 'My Blood': Listen
The footage includes drummer Josh Dun describing some local history about the glaciers and streams, the steep punishments for tearing up moss from rocks and facts about Will Smith's set-adjacent Icelandic retreat, all of which seem plausible, but also totally made up. All the while, Joseph is run through his rehearsal paces as his boots fill with water and storyboards pop up to illustrate the action.
So far the band has released the trilogy of interconnected tunes "Jumpsuit," "Nico and the Niners" and "Levitate," as well as the funk-influenced "My Blood." Trench is due out on Oct. 5 on Fueled by Ramen.
Watch the behind-the-scenes video — and the finished product — below.

redpajama_set_name: RedPajamaCommonCrawl


2 new official 20 century fox DVDs with Susan!
Jane Froman on "What's My Line?"
Smash Up The Story Of A Woman/Watch It Now!
2 forthcoming Susan Hayward DVDs!
BACK...FROM...THE...DEAD...or is it...AMONG THE LIVING..??
Life Imitating Art? Or Susan's Life?
When did you first meet Susan Hayward?
No further comments on quality DVD of "With a song in my heart"
I got the DVD of "With a song in my heart"
Isn't this a great note!
Susan Hayward new DVDs : Release dates...yes it is true!
A Reminder of "Back Street"
Re: Re: A Reminder of "Back Street"
I love this new lay-out.
HAPPY 90TH BIRTHDAY DEAR SUSAN.
Susan's singing in "I'll Cry Tomorrow"
Jill, you heard Osbourne live; I almost did!
Amazon.com Susan DVD releases announcements! true? More infos?
Happy Valentine's Day From Ginger!
Susan Hayward/Yourtube clip/I'll Cry Tomorrow/"Joe"
I know this will make you cry.. laugh.. and just downright grateful!


Fully restored 1871 Second Empire Victorian mansion, nestled in 33 acres of gardens, rolling meadows, ponds and woodlands. Original antiques that fill the parlors, library, dining room and sitting room. Play the 1869 Steinway square grand piano featured in the magazine 'Steinway'. Savor the solitude of natural surroundings or enjoy beautiful architecture inside including the famed "flying staircase". Stretch out on the grand porch or hide away on the romantic second floor balcony. Afternoon treats with tea or your favorite beverage are complementary upon check-in. Relish staying where Samuel Clemens chose to be a frequent overnight guest. We offer eight guest rooms, all with a private bath in the rooms. The Rosewood Room offers the "most expensive bed" in the state of Missouri you can sleep in. The John Garth Room has the best view of the gardens, and Avie's Room is best described as beautiful! Always rated as Missouri favorite B&B.
Lots of amenities including walking trails, paths, whirlpool tubs and hot tubs. Come feed the llamas and alpacas some carrots and they'll you forever. Even bicycles you may borrow.


Lumin is Now Sold Out!
Need we say more! We're just two weeks out from the grand opening of the new Snowmass Base Village! Join us for hot piping cocoa and treats at the Snowmass s'mores cart. Check out a free concert featuring Karl Denson's Tiny Universe, rock climbing exhibitions, free ice skating and more! See details by clicking here.
That's right! We're less than 8 weeks away from the Grand Opening of the new Snowmass Base Village! Join us 12-15-18! Concerts, food, libations, ice skating, rock climbing and more! Interested in a tour first-hand or how you can call Base Village home? Call us today at 970.924.9100 or email us at info@allinsnowmass.com.
We're just 16 weeks away from the grand opening of the new Snowmass Base Village. Make sure to mark your calendars for Dec. 15, 2018 and stay tuned here for details on what is sure to be a great party for locals and guests alike.
Watch as more than 70 community members celebrate the start of construction for One Snowmass.


text:
Grab secures $2B 5-year term loan facility as industry heats up in SEA
|In Investments, News
|By J. Angelo Racoma
Southeast Asian tech unicorn Grab has just announced that it has raised $2 billion via term loan facility from undisclosed institutional investors. The tech giant plans to use the facility to "strengthen its liquidity" and "diversify its financing sources."
The term loan is structured as a five-year term loan B that was upscaled from its initial principal amount of $750 million. Grab's press release also noted that the interest margin rate was lowered by 100 basis points from its former rate, 450 basis points above LIBOR.
It also announced that Moody's Investor Services and S&P Global Ratings issued to Grab ratings of B3 and B-, respectively. This makes Grab the first independently-rated tech company in Southeast Asia.
Grab plans to use the loan to strengthen its position as the leading super app in Southeast Asia. The loan will allow it to do two things, according to Grab's statement:
Strengthen its liquidity – The proceeds from the term loan will further enhance Grab's well capitalized position, and support its healthy trajectory, as it continues to strengthen its super app ecosystem to support Southeast Asia's daily essential consumer needs.
Diversify its financing sources – Since its founding, Grab has enjoyed significant support from its shareholders. The term loan facility will help broaden Grab's sources of financing and establish a long-term, diversified capital structure.
Southeast Asian startups have been working to become formidable competitors to Grab. Indonesian unicorn Gojek has recently secured $150 million through a deal with Indonesian telecom operator Telkomsel and plans to use this fund to expand its operations outside of Indonesia.
With tech companies seeing an even larger boom in its industry after 2020, Reuters has reported that Grab has plans for a US IPO sometime this year. The company has also announced that its fintech arm has raised over $300 million in a Series A funding round led by South Korean Hanwha Asset Management, K3 Ventures, GGV Capital, Arbor Ventures, and Flourish Ventures. Grab has also recently announced that the Monetary Authority of Singapore (MAS) has given Grab, in partnership with Singtel, a license for digital banking in Singapore.
CEO and Co-Founder of Grab Anthony Tan said, "I am deeply encouraged by the trust placed in us by investors who believe in our mission and recognise the value of our super app platform, as we continue making consistent progress in achieving our growth and sustainability milestones. With their support, we will invest in building a long-lasting, multi-local services business, so that millions of Southeast Asians can support their families and improve their lives with our everyday services."
JP Morgan acted as lead bookrunner for the loan, with Barclays, Deutsche Bank, HSBC, Mizuho, MUFG, and Standard Chartered as joint bookrunners.


text:
Hands-on harp session at Ely school
Harpist Bonnie Scott plays at Spring Meadow, Ely.
Children at Spring Meadow infant and nursery school were treated to a musical interlude when 15 year old Harpist Bonnie Scott held a "hands on harps" session.
Bonnie gave a live performance on a large gold harp, known as "Golden Boy", which is only used in special occasions, due to it being more than 100 years old.
She said: "Watching the children work their ways around Strawberry, Blueberry, BlackBerry and Minstrel brought back memories of when I started playing around nine years old.
"It was lovely to hear everyone make the loudest glissandi possible and learn the names of the F and C strings.
"Hopefully it was just as fun for the children as it was for me!"
She then ran a hands-on harp experience letting the children play on lap harps.
Bonnie has just finished her GCSE's at Witchford village college, and is planning on taking music at Hills Road sixth form college in September.
Since starting to play the harp in primary school, Bonnie has won competitions for playing and composing.
She has been an Aldeburgh young musician for the last three years and been accepted at the junior Royal Academy of Music in London, which starts in September.


text:
Happy 138th Birthday Fernand Léger
February 4, 2019 Scott Parker-Anderson Uncategorized Leave a comment
Today is the 138th birthday of the French artist Fernand Léger. His developed style of painting is distinctively his own. I see a combination of Picasso and Rivera cubism and the linear Art Deco formality. The world is a better place because he was in it and still feels the loss that he has left.
NAME: Fernand Léger
OCCUPATION: Painter
BIRTH DATE: February 4, 1881
DEATH DATE: August 17, 1955
EDUCATION: Paris School of Decorative Arts
PLACE OF BIRTH: Argentan, France
PLACE OF DEATH: Gif-sur-Yvette, France
BEST KNOWN FOR: French painter Fernand Léger created the abstract painting series "Contrast of Forms." His work blended elements of Cubism with his own unique style, "tubism."
Fernand Léger was born to a peasant family in the rural town of Argentan, France, on February 4, 1881. Léger's father was a cattle dealer who hoped his son would follow in his footsteps and choose what he deemed a practical trade. Although Léger was initially discouraged from becoming an artist, his father became supportive once he recognized Léger's gift for drawing.
With his father's approval, Léger enrolled in architecture school and accepted an apprenticeship under an architect in Caen. In 1901, upon completion of his two-year internship, Léger moved to Paris, France, where he worked as an architectural draftsman.
Wishing to further pursue his art education, Léger applied to the prestigious École des Beux-Arts and was unfortunately rejected.In 1903 he stated attending the Paris School of Decorative Arts instead, while also being unofficially mentored by two École des Beux-Arts professors who recognized his potential. Up until this point, Léger's painting style blended Impressionism with Fauvism. In 1907 he attended a retrospective of Paul Cézanne's work. From then on, Léger's work took on more elements of Cubism, but with his own unique style of slicing forms into tubular cylinders, casually referred to as "tubism."
In 1913, he started a series of abstract paintings called "Contrast of Forms." A year later, he put his art career on hold to serve in the French army during World War I. In 1916, he was gassed at Verdun. Having incurred a head injury, he was sent home and hospitalized until 1917.
After the war, Léger continued to paint but also tried his hand at other mediums, including book illustrations and set and costume designs for the theater. In 1924, Léger ventured to make his first film, Ballet Mécanique. That same year, he opened his own school of modern art.
As Léger's work matured in the 1920s and '30s, he increasingly incorporated elements of modernism—particularly representations of machinery and human figures expressing speed and movement. His notable paintings from this period include "The Mechanic," "Mona Lisa with Keys," "Adam and Eve," and "Composition with Two Parrots," among others.
With the arrival of World War II, in 1940, Léger temporarily relocated to America. During this time, he produced a series of paintings called "Divers," noted for its unique use of large patches of color that overlapped outlines to portrayed stylized figures of swimmers diving off docks in Marseille. This series was followed by two others also portraying human figures in motion: "Acrobats" and "Cyclists." In 1946, Léger went back to France, where he revitalized his art school and became active in the Communist Party. In the 1950s, Léger's work focused on the theme of the common man, and further expanded to include tapestry, pottery, stained glass and mosaics.
Léger died on August 17, 1955, in Gif-sur-Yvette, France.
61200 Argentan, France
artartistFernand LégerPainting
Previous Post: The Maltese Falcon – Required Viewing
Next Post: The Goonies – Required Viewing


We left the house at around 14:50, arrived "Bahrain International Airport" at 15:40, almost on time. Our plane was set to depart at 17:30, but gotten delayed and departed at around 17:50. This is because it was "Maghreb" prayer time and the praying room was very small and crowded, thus, a lot of people didn't board the plane on time.
We arrived to "Jeddah Airport" at around 20:10. We waited for around 10 minutes for the plane to park, another 20 minutes waiting for passport check in. We were done at around20:40.
At 22:30 we got in to the bus waiting to be taken to the hotel. At 22:50 the bus took off and we arrived to the hotel at 00:50. We settled in and rested a bit; found our luggage already set in our rooms (very well organized part from al-Tawheed Hajj services and Umrah as we didn't have to wait for them at the airport since they were sent by land a week prior to our travelling date!).
It's almost 1 am now, and we were notified to be prepared to start "Umrah" rites (manasik), therefore, we got in the bus again at 01:50. We arrived to the Kaaba and started Tawaf al-Qudoom ("arrival tawaf") at around 02:45.
Since it was very crowded, we obviously had to split up, so I ended up pairing with my brother. At 04:00 we finished the Tawaf, by then it was time for the first Fajer azan, so until I try to leave the crowd and look for a place to pray the 2 Rakaʿahs after the Tawaf, I met the rest of my family to start Sa'y (going back and forth between the two hills of As-Safaa and Al-Marwaa). After Fajer prayer, at around 05:40 we started the Sa'y, as we finished and got in the bus, it was around 07:00. 30 minutes later, we arrived to the hotel and went to eat breakfast. At around 08:30 we went to sleep (mind you that the day we went to travel I woke up at 8:30, and until I finished the Umrah rites, I did not have any sleep!).


Daniel is the Vice President of the Smart Home business within Amazon Devices division. Prior to this role Daniel led product, marketing, and customer engagement teams for two businesses, Dash button and DRS, and Prime Exclusive phones. He also led Planning and Analytics for Amazon Devices, including market research, pricing, and demand and supply planning for all Amazon devices worldwide. Daniel joined Amazon in 2009 as the Director of the higher-education segment business for Amazon Devices where he focused on enabling students and educators to use Amazon devices and apps for reading and studying. Prior to joining Amazon, Daniel was COO of visiblebody.com, an award-winning medical and scientific education company located in the Boston area; he and his partners spun that out of Argosy, an educational technology and curriculum development firm serving all major US textbook publishers, where he was also COO.


Eligibility: Contest open to legal residents of the 50 United States and the District of Columbia who are at least 18 years of age as of February 15, 2019. You may not enter if you are a professional children's book illustrator or author; or if you are represented by a book publishing agent; or if any of your work has been previously published by any book publisher (self-published authors may enter). Employees of Hachette Book Group, Inc. ("Sponsor"), and its parents, subsidiaries, affiliates, retailers, suppliers, distributors, advertising/promotion agencies (collectively, "Promotion Entities") and their immediate family members (spouses, mothers, fathers, brothers, sisters, sons, daughters and their respective spouses, regardless of where they reside) and/or those living in the same household of such employees, whether or not related, are not eligible to enter or win. Contest is void where prohibited by law.
Submission Period: The Submission Period for all entries is between 12:01 am Eastern Time ("ET") on February 15, 2019 and 11:59 pm ET on June 7, 2019.
How to Enter: You may enter by visiting http://www.LBArtistAward.com and (i) filling out a submission form with your name, mailing address (no P.O. boxes), phone number, date of birth and a valid email address where Sponsor may contact you, and (ii) uploading your Submission (defined below). Proof of Submission is not considered proof of delivery or receipt of such Submission.
To encourage the development of high-quality children's picture books that resonate with readers of diverse backgrounds and experience, that in some manner draws from the rich cultural experience of this country—whether it manifests in character, theme, setting, plot, or is derived simply from the artist's own experience of identity. Diversity includes literal or metaphorical inclusion of characters of underrepresented ethnicity, religious background, gender identity, class, mental or physical disability, or any other non-dominant populations.
Is written in English and appropriate for children up to 8 years of age.
A "Portfolio" of at least 6 additional pieces of sample art that is representative of your work (unrelated to the picture book mockup).
Must be named using the following format: "firstname_lastname.pdf".
Additional 6 pieces of sample portfolio art should be included at the end of the picture book dummy, and be all part of the same uploaded file.
No team or collaborative Submissions.
Sponsor reserves the right, in sole discretion, to disqualify any Submission if it violates or potentially violates any of the above Restrictions, or any third party's rights, or if Sponsor deems the Submission to be lacking in taste or quality.
Use of Submissions: Sponsor may post select excerpts of the winning Submission (excluding anything from the Portfolio) along with the entrant's name and/or entrant's state of residence on any of its social media pages, accounts and/or websites, including, but not limited to: http://www.LBArtistAward.com, LB-Kids.com, LittleBrownLibrary.com, Twitter.com/LBKids, Twitter.com/LBSchool,Facebook.com/LittleBrownBooks,Facebook.com/LittleBrownSchool, Pinterest.com/LBKids, Instagram.com/LB_Kids, LBSchool.tumblr.com. Sponsor has no obligation to make use of the rights granted herein.
Judges' decisions will be final and binding in all matters related to this Contest. In the event of a tie, the judges will reconsider the tied Submissions using the judging criteria stated above to determine one (1) Award winner. Sponsor reserves the right to not provide the Award if, in its sole discretion, it does not receive a sufficient number of eligible and qualified Submissions. Sponsor will not reveal the judging scores for any Submission.
Winner Notification: The potential winner will be notified by telephone, email, and/or mail using the contact information provided at the time of entry. Sponsor shall have no liability for any winner notification that is lost, intercepted, delayed, or not received by the potential winner for any reason. The potential winner will be required to execute and return any reasonably required documentation including an Affidavit of Eligibility, Liability Waiver and, unless prohibited, a Publicity Release ("Award Claim Documents") within fourteen (14) days of notification and receipt of Award Claim Documents from Sponsor. If a potential winner does not respond to the initial notification within the time period stated, all required Award Claim Documents are not returned within the specified time period (if applicable), award notification is returned as undeliverable, or a potential winner is not in compliance with these rules, the applicable award will be forfeited and, at Sponsor's discretion, an alternate winner may be selected.
An opportunity for the Submission to be reviewed by the LBYR's editorial team for possible future publication. (**A subsequent book publishing deal, if any, with Sponsor will be separately negotiated between the winner and Sponsor, and is not considered part of the Award).
The winner will receive round-trip travel (coach class, air or ground transportation) from the winner's residence in one of the 50 United States or District of Columbia to New York, New York. The winner will stay one (1) night at a hotel (a standard, single occupancy room) in New York City. The trip will be scheduled between October 1, 2019 - February 1, 2020. The trip must be taken during those dates, or as modified by Sponsor in its sole discretion. If the winner is not available on the specific dates, the winner will forfeit the Award, and the Award may be awarded to an alternate winner.
The Approximate Retail Value ("ARV") of the Award is four thousand seven hundred ($4,700) dollars. Actual value of the Award may vary depending, for example, on the location of winner's residence. All travel, hotel and other arrangements must be made through Sponsor. The choice of airline, hotel, and type of room will be subject to Sponsor's decision in its sole discretion. Air transportation does not include government taxes, passenger facility or other government, and incidental airline fees (e.g. baggage, food etc.). All such fees must be paid by the winner and are due at time of confirmation of the flight dates; once paid these are non-refundable. Flight insurance, travel insurance, gratuities, personal expenses (including food) and all other expenses not specified as part of the Award and are the sole responsibility of the winner. Other travel/accommodation restrictions may apply.
Air and/or ground transportation tickets are not valid for upgrades and/or frequent flyer miles. All tickets are subject to the vagaries of flight variation, work stoppages, and schedule or route changes. In the event a winner engages in behavior that (as determined by Sponsor in Sponsor's sole discretion) is obnoxious or threatening, illegal or that is intended to annoy, abuse, threaten or harass any other person, Sponsor reserves the right to terminate the trip early and send the winner home with no further compensation.
The winner will be responsible for all taxes that may become payable as a result of the Award (federal, state, city, sales), as well as the proper reporting or declaration of income and payment of taxes and related costs, including interest and/or penalties, to the appropriate governmental authority(ies).
Right of First Offer: Sponsor has the right, within three (3) months of the prize announcement, to make a first offer to the winner for exclusive, world, all languages publishing rights to the Submission. Sponsor has no obligation to make any offer, and no obligation to use the winning submission for any purpose.
General Award Conditions: Certain additional terms and conditions apply to use of the American Express® gift cards (or another gift card, if applicable, see below). The Award will be awarded only if the potential winner fully complies with these Official Rules. No interest will be paid on any Award. The Award is non-transferable and non-assignable and non-cash portions of the Award cannot be redeemed for cash. All details and other restrictions of the Award not specified in these Official Rules will be determined by Sponsor in its sole discretion. Sponsor reserves the right to substitute a different gift card due to any reason whatsoever. The Award is awarded "as is" with no warranty or guarantee, either express or implied.
Publicity: By participating in the Contest, each entrant irrevocably grants the Promotion Entities and their respective successors, assigns and licensee, the right to use such entrant's first name, state of residence, and select portions of the Submission (excluding the Portfolio) for advertising and promotional purposes, as well as in, on or in connection with the Contest and/or other promotions conducted by the Promotion Entities, and each entrant hereby releases the Promotion Entities from any liability with respect thereto. Promotion Entities are not responsible for any unauthorized use of Submissions by third parties. Sponsor has no obligation to make use of the rights granted herein.
General Liability Release/Force Majeure: Entrants agree that the Promotion Entities (a) shall not be responsible or liable for any losses, damages or injuries of any kind resulting from participation in the Contest or any Contest-related activity, or from entrant's acceptance, receipt, possession and/or misuse of the Award, and (b) have not made any warranty, representation or guarantee express or implied, in fact or law, with respect to the Award, including, without limitation, to such Award's quality or fitness for a particular purpose. Promotion Entities assume no responsibility for entrant's participating in the Contest or use of the Award. Promotion Entities are not responsible for (i) lost, late, incomplete, damaged, inaccurate, stolen, delayed, misdirected, undelivered or garbled Submissions or email or entry forms; (ii) lost, interrupted or unavailable network, server, Internet Service Provider (ISP), website, or other connections, availability or accessibility or miscommunications; (iii) failed computer, satellite, telephone or cable transmissions, lines, or technical failure; (iv) jumbled, scrambled, delayed, or misdirected transmissions or computer hardware or software malfunctions, failures or difficulties; (v) other errors or difficulties of any kind, whether human, mechanical, electronic, computer, mobile device, network, typographical, printing or otherwise, relating to or in connection with this Contest, including, without limitation, errors in these Official Rules or errors or difficulties which may occur in connection with the administration of the Contest, the processing or judging of Submissions, the announcement of the Award or in any Contest-related materials; (vi) any incorrect or inaccurate information, whether caused by users, tampering, hacking, or by any equipment or programming associated with or utilized in the Contest; or (vii) injury or damage to any person's computer related to or resulting from participating in this Contest. Sponsor may, in its sole discretion, disqualify any individual who is found to be tampering with the entry process or the operation of the Contest, to be acting in violation of these Official Rules, or to be acting in an unsportsman-like or disruptive manner, or with the intent to disrupt or undermine the legitimate operation of the Contest, or to annoy, abuse, threaten or harass any person, and Sponsor reserves the right to seek damages (including attorneys' fees) and other remedies from any such person the fullest extent permitted by law.
In the event Sponsor is prevented from awarding the Award or continuing with the Contest as contemplated herein by any event beyond its control, including but not limited to fire, flood, natural or man-made epidemic, earthquake, explosion, labor dispute or strike, act of God or public enemy, satellite or equipment failure, riot or civil disturbance, terrorist threat or activity, war (declared or undeclared) or any federal state or local governing law, order, regulation, public health crisis (e.g. Ebola), order of any court or jurisdiction, or other cause not reasonably within Sponsor's control (each a "Force Majeure" event or occurrence), then subject to any governmental approval as may be required, Sponsor shall have the right to modify, suspend, or terminate the Contest. If the Contest is terminated for Force Majeure before the designated end date, Sponsor will (if possible) select the winner from all eligible, non-suspect Submissions received as of the date of the event of Force Majeure giving rise to the termination using the judging procedure set forth above or as otherwise deemed fair and appropriate by Sponsor.
These Official Rules cannot be modified or amended in any way except in a written document issued in accordance with law by a duly authorized representative of Sponsor. The invalidity or unenforceability of any provision of these rules shall not affect the validity or enforceability of any other provision. In the event that any provision is determined to be invalid or otherwise unenforceable or illegal, these Official Rules shall otherwise remain in effect and shall be construed in accordance with their terms as if the invalid or illegal provision were not contained herein.
Governing Law/Jurisdiction: ALL ISSUES AND QUESTIONS CONCERNING THE CONSTRUCTION, VALIDITY, INTERPRETATION AND ENFORCEABLITIY OF THESE OFFICIAL RULES OR THE RIGHTS AND OBLIGATIONS OF ENTRANTS OR SPONSOR IN CONNECTION WITH THE CONTEST SHALL BE GOVERNED BY AND CONSTRUED IN ACCORDANCE WITH THE INTERNAL LAWS OF THE STATE OF NEW YORK WITHOUT GIVING EFFECT TO ANY CHOICE OF LAW OR CONFLICT OF LAW RULES OR PROVISIONS THAT WOULD CAUSE THE APPLICATION OF ANY OTHER STATE'S LAWS.
Winners List: To obtain a copy of any legally-required Winners List, send a self-addressed, stamped envelope to Little, Brown Emerging Artist Award Winners List, Hachette Book Group, Inc., 1290 Avenue of the Americas, New York, NY 10104. Attn: LBYR Art Department, NB-Work-Station #5-589. All requests must be received by Sponsor within six (6) weeks after the end of the Submission Period.
Sponsor: Hachette Book Group, Inc., 1290 Avenue of the Americas, New York, NY 10104.
American Express is a registered trademark of American Express Travel Related Services Company, Inc. This Contest is not sponsored, endorsed or administered by American Express Travel Related Services Company, Inc.


We sure hit the ground running this week. Since returning from Spring Break, we've already had an awesome end of the year Show Choir concert, plus a great clinic with guest clinician Matt Greer in Concert Choir today, and two fantastic Show Choir Audition Workshops with Jonathan Ragsdale. Phew! On the horizon we still have State Choir as well as preparations for our Spring Choir Concert followed by our end of the year Banquet and Graduation. The end of the year will be here before we know it.
Due to the high number of students participating in Rube Goldberg Day, all Zero Hour activities in the Music Building are CANCELLED.
This is with the exception of auditions which will be held after school on May 2, 3, and 4.
Busses depart from EHS at approximately 10:45 and will return around 4:45 pm. Students will receive an excused Eagle absence for periods 6 & 7 on that day. Students will be provided lunch while at Sue Cleveland. All students should wear concert appropriate attire, black shoes (no tennis shoes) and will pick up robes at school before we depart.
The Albuquerque Theater Guild has sponsored our group's attendance at the April 29th performance of UNM's spring musical. Our assistant, Gigi Guajardo is performing in this show and has facilitated this awesome group event. There is no cost for attendance and students will be bussed from EHS to UNM for the event. Sign ups are now closed for this event. There will be more details to follow. This show does have a PG-13 rating.
Mark your calendars for this year's banquet! It is an event not to be missed! Saturday, April 30th, we will be celebrating this amazing year of choir. Emails have been sent out with information regarding purchase of tickets and the deadline for ticket orders is APRIL 15th! Remember that your choir student has already paid for their ticket as part of the events fee.


Six issues of our award successful magazine and day by day supply of The Download, our e-newsletter of what is essential in technology and innovation. Its voice assistant know-how is the Siri of China, and its actual-time moveable translator places AI to outstanding use, overcoming dialect, slang, and background noise to translate between Chinese and a dozen other languages with shocking accuracy.
Before your subscription expires Amazon will notify you by way of e-mail of the rate at which you will renew. Every day, we provide an clever, lucid, and authoritative filter for the overwhelming flood of details about technology. Transferring to include AI into its businesses because it focuses on technological innovation in wind and renewable power, information-pushed providers, and other business traces.
Reporting on vital applied sciences and innovators since 1899, with the backing of the world's #1 know-how institute. Minting cash selling its widespread cell phones and laptops whereas including spectacular names to its AI research staff and promising to do extra manufacturing within the U.S.
Editorial and publication insurance policies: NAfME editors will edit reviews to make sure conformity with the reviewers' pointers and MEJ style, which relies on The Chicago Manual of Fashion (fifteenth ed.), printed by the University of Chicago Press.
Number of pure-language data sets constructed into the company's AI research tool, ParlAI. I would like to obtain information about other MIT Know-how Evaluate initiatives. Writing the evaluation: Learn the e-book in its entirety, or spend a while using the software program.


Certified Professional in Healthcare Quality CPHQ course in New York is created for specialists in the field of healthcare.
CPHQ training in New York is provided by Anexas, the No. 1 Healthcare Quality Course provider in New York . Anexas is one of the top training service providers in the country. We provide Certified Professional in Healthcare Quality Training according to the current industry standards.