\section{Introduction}
Science requires reproducibility, but many sub-fields of science have recently experienced a reproducibility crisis, eroding trust in processes and results and potentially influencing the rising rates of scientific retractions \cite{baker_1500_2016,cokol_retraction_2008,vasilevsky_reproducibility_2013}.
Reproducibility is also critical for machine learning research, whose goal is to develop algorithms to reliably solve complex tasks at scale, with limited or no human supervision. Failure of a machine learning system to consistently replicate an intended behavior in a context different from which that behavior was defined may result in dramatic, even fatal, consequences~\cite{levin_tesla_2018}.
Ranking prominently among machine learning applications that may put human lives at stake are those related to Machine Learning for Health (\textsc{ML4H}\xspace). In a field where applications are meant to directly affect human health, findings should undergo heavy scrutiny along the validation pipeline from research findings to applications deployed in the wild. For example, in 2018, 12 AI tools using \textsc{ML4H}\xspace algorithms to inform medical diagnosis and treatment were cleared by Food and Drug Administration (FDA) and will be marketed to and potentially used by millions of Americans~\cite{muoio_roundup:_2018}. Verifying the reproducibility of the claims put forward by the device manufacturer should thus be a main priority of regulatory bodies~\cite{parikh2019regulation}, extending the need for reproducible \textsc{ML4H}\xspace results beyond the machine learning research community.

Unfortunately, several  factors relating to the availability, quality, and consistency of clinical or biomedical data make reproducibility especially challenging in \textsc{ML4H}\xspace applications.
In this work, we make several contributions. First, we present a taxonomy of reproducibility tailored to \textsc{ML4H}\xspace applications, and designed to capture reproducibility goals more broadly. Second, we use this taxonomy to define several metrics geared towards quantifying the particular challenges in reproducibility faced within ML4H, and conduct a comprehensive review of the published literature to support our claims and compare \textsc{ML4H}\xspace to machine learning more generally. Finally, we build on this analysis by exploring promising areas of further research for reproducibility in \textsc{ML4H}\xspace.

\section{A Reproducibility Taxonomy}
The common understanding of reproducibility in machine learning can be summed up as follows:
A machine learning study is \textit{reproducible} if readers can fully replicate the exact results reported in the paper.
We will call this concept \textit{technical replicability}, as it is centrally concerned with whether or not one can replicate the precise, technical results of a paper under identical conditions. Though intuitive, we argue technical replicability is actually only a small part of the goal of ``reproducibility'' more generally. This discrepancy has been noted historically in various ways \cite{drummond_replicability_2009,gundersen_2018,joelle_pineau_reproducible_2018,plesser_reproducibility_2018} and is made apparent by the use of the term in the natural and social sciences, where attempted reproductions will often occur at different labs, using different equipment/staff, etc.
We argue that in order for a study to be fully reproducible, it must meet three tiers of replicability:

\SetLabelAlign{parright}{\parbox[t]{\labelwidth}{\raggedleft#1}}
\begin{description}[
leftmargin=!,labelwidth=\widthof{\bfseries Conceptual Replicability},align=parright
]
    \item[Technical Replicability] Can results be replicated under \emph{technically} identical conditions?
    \item[Statistical Replicability] Can results be replicated under \emph{statistically} identical conditions?
    \item[Conceptual Replicability] Can results be replicated under \emph{conceptually} identical conditions
\end{description}

\emph{Technical replicability} refers to the ability of a result to be fully technically replicated, yielding the precise results reported in the paper. For example, this entails aspects of reproducibility related to code and dataset release. \emph{Statistical replicability} refers to the ability of a result to hold under re-sampled conditions that yield different technical configurations, but should not statistically affect the claimed result (e.g., a different set of random seeds, or train/test splits). Note this is related, but not identical, to the notion of \emph{internal validity}~\cite{campbell1986relabeling} commonly used in social science research. Similarly, \emph{conceptual replicability} is heavily related to \emph{external validity}~\cite{pearl2014external}, as it describes the notion of how well the desired results reproduces under conditions that mach the conceptual description of the purported effect. Note that this is task-definition dependent; claiming a task has a greater conceptual horizon of generalizability makes it harder to satisfy this reproducibility requirement.

All three of these replicability criteria are central for full reproducibility: without technical replicability, one's result cannot be demonstrated. Without statistical replicability, one's result will not reproduce under increased sampling and the presence of real-world variance. And lastly, without conceptual replicability, one's result does not depend on the desired properties of the data, but instead depends on potentially unobserved aspects of the data generation mechanism that, critically, \emph{will not reproduce when deployed in practice}.

Under each of these lenses, ML4H differs from general machine learning domains in critical ways and presents unique challenges.

\section{Core Reproducibility Challenges in ML4H}
In this section, we illustrate both through qualitative arguments and a quantitative literature review that ML4H lags behind other subfields of machine learning on various reproducibility metrics. Our literature review procedure entailed manually extracting and annotating over 300 papers from various venues, spanning ML4H, Natural Language Processing (NLP), Computer Vision (CV), and general machine learning (general ML).\footnote{NLP and CV were chosen to represent broad fields with a significant applied focus, much like ML4H. General ML was chosen to have an unbiased comparison to the field more broadly.} The full procedure used for this statistical review is detailed in the Appendix (Section~\ref{sec:review_procedure}), and final quantitative results on several key metrics can be seen in Figure~\ref{fig:paper_stats}, though they are also detailed in the text where appropriate. The full set of results is available at https://github.com/evidation-health/ICLR2019.

\begin{figure}[t]
    \centering
    \includegraphics[width=1\textwidth]{paper_stats.pdf}
    \caption{Fraction of papers satisfying certain conditions by ML field. See the Appendix (Section~\ref{sec:review_procedure}) for detailed descriptions of the underlying data collection procedure. Note that ML4H consistently lags other subfields of machine learning on all measures of reproducibility save inclusion of proper statistical variance.
    }
    \label{fig:paper_stats}
\end{figure}

\subsection{Technical Replicability}
ML4H faces several key challenges with regards to technical replicability. Firstly, health data is privacy sensitive, making it difficult to release openly without either incurring risks of re-identification, or severely diminishing utility by applying aggressive de-identification techniques~\cite{dwork_ullman_2018}. As a result, few public datasets are available, and those that are available are used extremely frequently, leading to a risk of dataset-specific over-fitting. To this point, approximately only 51\% of the ML4H papers we examined used public datasets, as compared to over 90\% of both CV and NLP papers, and approximately 77\% of general ML papers.

ML4H scores even more poorly when it comes to code release, preprocessing specification, and cohort description; only approximately 13\% of the papers we analyzed released their code publicly, compared to approximately 37\% in CV and 50\% in NLP. A previous study that reviewed the prevalence of code release in AI reported that only 6\% of papers released code \cite{gundersen_2018}, which is lower than all of our estimates. This discrepancy may be due to sampling papers from different conferences and different time periods (2013-2016 vs. 2018 for the current study). Specific prior works have also already examined the prevalence of code release and dataset cohort sub-selection within ML4H, finding that even when restricting focus to public datasets, studies often do not release code or include sufficiently informative text to enable a full technical reproduction \cite{hegselmann_reproducible_2018,johnson2017reproducibility}.  Note that code release is itself not necessarily sufficient for full technical replicability; even when code is released, it may not run correctly, it may exclude critical details, or it may fail to generate the results reported in the paper.

\subsection{Statistical Replicability}
To assess the state of statistical replicability in ML4H, we quantified how often papers describe the variance around their results (e.g., by listing both the mean and the standard deviation of a performance metric over several random splits). Interestingly, while ML4H's rate of this is still relatively low (approximately 38\%), it is higher than that of CV, NLP, or the general domain (17\%, 33\%, and 32\%, respectively).

Though this is an encouraging sign, we feel there is still significant room for improvement.
Even in other fields of machine learning, with arguably less complex data types, repeated studies have shown that published models fail to statistically reproduce when appropriate statistical procedures are implemented and fair hyperparameter search/preprocessing methods are used: in \cite{recht_imagenet_2018}, researchers find that published models evaluated using the public ImageNet test set show consistent performance drops when trained/tested on other random splits within ImageNet, and in \cite{melis_state_2017,lucic_are_2017}, researchers find that published performance deltas between various model architectures for language models and generative adversarial networks, respectively, fail to persist under more robust hyperparameter search and statistical comparison techniques. In \cite{henderson_deep_2017}, researchers examine practices in deep reinforcement learning which can limit or enhance studies' reproducibility. Due to our inability to technically replicate many ML4H works given dataset and code release issues, we cannot assess the extent of these issues to the same degree in ML4H. However, it seems doubtless that similar problems affect our field, perhaps to an even greater degree given that our datasets are frequently relatively small, high dimensional, sparse/irregularly sampled, and suffer from high rates of noise.

\subsection{Conceptual Replicability}
The critical issue in ML4H preventing conceptual replicability is the lack of multi-institution datasets in healthcare and the lack of usage of those that do exist. Whereas approximately 83\% of CV studies and approximately 66\% of NLP studies used multiple datasets to establish their results, only approximately 19\% of ML4H studies did. Using only a single dataset is potentially dangerous, as it is known that developing ML4H models that generalize over changing care practices or data formats is challenging. In \cite{gong_predicting_2017}, researchers demonstrate the dangers of training a model on raw data from one institution and transferring to another (using a simulated institutional divide within the MIMIC dataset), and in \cite{nestor_rethinking_2018}, researchers establish that without using manually engineered representations, ML4H models exhibit significant degradation in performance over time as care patterns evolve. These results are not surprising; health data is rife with hidden confounders, differs significantly between data collection and deployment environments, drifts over time, and further differs in structure and concept between different healthcare institutions~\cite{caruana2015intelligible}.

\section{Opportunities for Improvement}
In this section, we present several practical suggestions for enhancing reproducibiltiy for ML4H from the perspective of three main \textsc{ML4H}\xspace stakeholders: the \textsc{ML4H}\xspace research community, data providers, and related journals and conferences.  Figure~\ref{fig:recommend} summarizes these suggestions.

\begin{figure}
    \centering
    \includegraphics[width=0.6\columnwidth]{recommendations.pdf}
    \caption{Summary of recommendations for different stakeholders.
    }
    \label{fig:recommend}
\end{figure}

\subsection{Create Shared Research Resources}
Data providers such as hospitals, clinical research centers, and government agencies produce vast amounts of valuable data. Unfortunately, as suggested by our literature review, few datasets are available for a wide range of researchers to explore. We call for more instances of large data trusts where medical institutions can anonymously pool data for researchers to use and create algorithms. Several prominent examples already exist for the field to draw on in creating such repositories, including MIMIC~\cite{johnson2016mimic}, the U.K and Japan Biobanks~\cite{nagai2017overview,sudlow2015uk}, eICU~\cite{pollard_eicu_2018}, the Temple University Hospital EEG Corpus~\cite{harati2014tuh}, and Physionet~\cite{goldberger2000physiobank}. This recommendation is especially compelling as corporate entities increasingly invest in the ML4H space, particularly with regards to data~\cite{rajkomar_scalable_2018,wood_improving_2019}. Anchoring our research progress to large scale datasets in non-public or non-academic hands poses a dangerous precedent, especially in light of recent patent filings~\cite{bresnick_google_2019,google_inc._system_nodate}.

\subsection{Integrate Multi-Institute Datasets}
Multi-institute datasets enable studies to assess their ability to translate to new contexts, a critically understudied facet of ML4H research. Recent strides have been taken in this domain with the release of the eICU dataset~\cite{pollard_eicu_2018}, one of ML4H's first large scale, multi-institution EHR datasets, and researchers are already analyzing how to form generalizable models using this resource \cite{johnson_generalizability_2018}. Lastly, Observational Health Data Sciences and Informatics (OHDSI)~\cite{hripcsak2015observational} provides a mechanism to run observational health studies across multiple institutions and countries.
We encourage more collaborative efforts in this area from data providers to improve conceptual replicability.

\subsection{Prospectively Collect Data}
Data collected as a by-product of care, then later released for research purposes (e.g., MIMIC \cite{johnson_mimic-iii_2016}), presents serious privacy risks and contains many confounding variables. The landscape of these privacy risks and the nature of the confounding variables change (though they do not necessarily lessen) if data is instead prospectively collected directly from new, consented participants. These types of data collection regimes are logistically challenging, but are possible.
The NIH's All of Us Research Program \cite{national_institutes_of_health_all_nodate}, Evidation's DiSCover Project \cite{evidation_health_digital_2018}, and Google's Project Baseline \cite{verily_why_2017}
are examples of these approaches.

\subsection{Adopt Rigorous Statistical Methods}
ML4H researchers should be more rigorous in the development, refinement, and dissemination of statistical best practices (e.g., the proper procedures for model comparisons, etc.). Holding ourselves to high standards of statistical rigor, potentially including periodic statistical audits of our own statistical replicability, will help ensure we mitigate the problems other fields have found with ``over-fitting via publication,'' e.g. \cite{recht_imagenet_2018}.  Several commonly-used challenges already exist in ML4H with fixed train/test sets (e.g., \cite{ghassemi_you_2018,stubbs_annotating_2015}) which could be used for these kind of post-hoc replication studies.

\subsection{Develop New Privacy-Preserving Analysis Techniques}
Technological solutions can also be employed to help mitigate privacy concerns. ML4H researchers can explore noised, fully- or partially- simulated, or encrypted datasets. In cases where data cannot be released, techniques to train distributed models without sharing data have been proposed~\cite{vepakomma2018split}. Synthetic data can be an excellent tool to help enable researchers to still meaningfully release their code with full end-to-end realizations of their pipeline, as is done in, e.g., \cite{zink_fair_2019}. Technology for producing synthetic patient-level data also already exists \cite{walonoski_synthea:_2018}.

\subsection{Propose Pre-Registration Alternatives}
In the biomedical sciences, ``observational studies,'' under which definition nearly all ML4H research falls, undergo intense scrutiny to ensure they are not susceptible to statistical artifacts---in particular, increasingly these studies are required to be \emph{pre-registered},\footnote{
Meaning they are required to report their goal and planned preprocessing/modeling scheme before they run any experiments in order to avoid intentional or unintentional statistical fraud. This idea is not without its detractors \cite{lash_commentary:_2012,editors_registration_2010}.
} a move promoted by scientists and major journals alike \cite{lancet_should_2010,loder_registration_2010,williams_registration_2010}. Such prospective checks make unintentional statistical fraud more difficult, but are utterly absent in ML4H. A verbatim application of the pre-registration practice in use in the epidemiology community is likely an impractical solution due to the intrinsic exploratory nature of model development, but other techniques, such as systematic re-release of new data and/or rotation of official train/test splits have been found to help reduce the presence of statistical errors in other fields. ML4H researchers and academic publishers should engage in serious conversation on what the best vehicle for these checks and balances are.

\subsection{Establish Reporting Requirements and Require Code Release}
If conferences and journals required greater rates of data/code release, or the use of additional data/code availability statements at publication, this would instill a significant pressure on the field to address some of the foundational barriers to reproducibility. Conferences and journals have the additional ability to insist on high standards for the reporting of statistical variance around results, hyperparameter search procedures, and evaluation mechanisms, each of which would help ensure that we maintain high standards of technical and statistical replicability.

\subsection{Develop Data Standards and Reporting Standards}
Collaborative efforts in developing data standards and reporting standards is another avenue for improving reproducibility. Healthcare analytics organizations have developed data standards like the OMOP standard \cite{overhage_validation_2012} or the FHIR standard \cite{hl7_introducing_2018}, but they are not commonly adopted in \textsc{ML4H}\xspace research. Increased use of standards would make it easier to technically and conceptually replicate \textsc{ML4H}\xspace studies. Similarly, when \textsc{ML4H}\xspace datasets are created, we need greater descriptions of their contents, potential confounders and biases, missing data prevalence and distribution, and how they were created. Increasing the use of ``specs'' or ``datasheets'' describing datasets would help allay these concerns---efforts in the broader machine learning community are also approaching this goal \cite{gebru_datasheets_2018} and we should vigorously adopt these practices in \textsc{ML4H}\xspace.

\section{Conclusion}
In this work, we have framed the question of reproducibility in \textsc{ML4H}\xspace around three foundational lenses: technical, statistical, and conceptual replicability. In each of these areas, we argue both qualitatively and quantitatively, through a manual, extensive review of the literature, that \textsc{ML4H}\xspace performs worse than other machine learning fields in several reproducibility metrics we have identified. While keeping in mind the intrinsic challenges of data acquisition and use that plague the field, we highlight several areas of opportunities for the future, focused around improving access to data, expanding our trajectory of statistical rigor, and increasing the use of multi-source data to better enable conceptual reproducibility.

\section*{Acknowledgments}
This paper benefited substantially from the help of many people. Most notably, Bret Nestor, Amy Lu, Denny Wu, Elena Sergevea, and Di Jin all helped annotate papers for our analysis.

Additionally, this work was funded in part by National Institutes of Health: National Institutes of Mental Health grant P50-MH106933, a Mitacs Globalink Research Award, as well as a CIFAR AI Chair at the Vector Institute, and an NSERC Discovery Grant.


\subsection{Dataset Statistics}

\subsection{Evaluation of the Annotation Method}
\label{sec:evalAnno}

For validating the accuracy of our  annotation method, we manually annotated the
3D locations  of the  3D joints in  randomly selected frames  of a  sequence, by
relying on  the consolidated point cloud  from the 5 cameras.   We then compared
these locations to the ones predicted with our method (explained in Section~\ref{sec:multiCameraSetup}) using the multi-camera setup.

As shown in the last column of Table~\ref{tab:table2}, our method achieves an
average joint error accuracy of lower than 8mm on average, with an Area Under the Curve
metric~(AUC) of 0.79. This metric is comparable with the results reported
for the recent FreiHAND dataset~\cite{zimmermann2019freihand}~(AUC=0.791). Note that the
occlusions in our dataset are higher due to larger objects and that we do not use green screens.

To analyze the influence of the different terms in Eq.~\eqref{eq:global}, we run
the optimization  of Eq.~\eqref{eq:global}  by enabling only  a subset  of these
terms, and  report the results  in Table~\ref{tab:table2}.  While  $E_\text{silh}$ and
$E_\text{dpt}$ terms alone  cannot provide good pose estimates,  together they provide
better estimates  as it leads  to a loss function  with less local  minima.  The
$E_\text{3D}$  term provides  a  minor improvement  in estimates  but  speeds up  the
convergence.  Though  the physical plausibility  term $E_\text{phy}$ does not  help in
improving the pose  estimates, it results in more natural  grasps.  The last two
columns  show the  effect of  multi-frame based  joint optimization  compared to
single-frame  based  optimization  when  all   the  terms  are  considered.  The
multi-frame  multi-camera based  optimization over  all the  terms improves  the
accuracy by about 15\%.

The  accuracy of  the single  camera based  annotation method  is calculated  by
considering the annotations from the multi-camera method as ground truth for a given
sequence. More specifically, for a sequence of
1000  frames, we compute the average difference   between  hand+object  mesh   vertices  obtained  from  single  and
multi-camera setups. Further, we calculate the  accuracy after each stage of the
single-camera  setup.  The  results  are given  in  Table~\ref{tab:table5}.  The
estimated poses  with these two  methods are consistent  with each other  with an
average mesh error  of 0.77cm and 0.45cm for hand  and object, respectively. The
final refinement stage yields a 15\% improvement in accuracy.

\begin{table}[]
  \begin{center}
    \resizebox{0.47\textwidth}{!}{%
      \begin{tabular}{lcccc}
	\toprule
	Method & \specialcell{Mesh Error$\downarrow$} & F@5mm$\uparrow$ & F@15mm$\uparrow$  & \specialcell{Joint Error$\downarrow$}\\
	\midrule
	Joints2D & 1.14 & 0.49 & 0.93 & 3.14 \\
	Joints2D + Dir. Vec. & \textbf{1.06} & \textbf{0.51} & \textbf{0.94} & \textbf{3.04} \\

	\cite{hasson2019learning}  & 1.10 & 0.46 & 0.93 & 3.18 \\
	\bottomrule
    \end{tabular}}
    \vspace{-2mm}
    \caption{Evaluation  of  different  methods   for  single  frame  hand  pose
      prediction. The Mesh Error (in cm) and \textit{F}-score are obtained after
      aligning the  predicted meshes  with ground truth  meshes. The  Mean joint
      error (in  cm) is obtained after  aligning the position of  the root joint
      and overall scale with the ground  truth. Hand pose prediction using joint
      direction  predictions along  with  2D joint  predictions provides  better
      accuracy    than   directly    predicting the   MANO    parameters   as    in
      \cite{hasson2019learning}.}
    \vspace{-6mm}
    \label{tab:tablePoseEst}
  \end{center}

\end{table}

\subsection{Evaluation of Hand Pose Prediction Method}

We  trained our  single frame  hand  pose prediction  method explained  in
Section~\ref{sec:handposeSingleFrameSec} on 66,034  frames from our \datasetname
dataset.  We evaluated it  on a test set of 13  sequences captured from different
viewpoints  and  totaling  11,524   frames.  The  test  set  sequences also  contain
subjects and objects  not present  in  the training  set.

We  report three  different metrics  from previous  works: Mean  joint position
error after aligning the position of the root joint and global scale with ground
truth~\cite{Zimmermann2017}; Mesh error measuring the average Euclidean distance
between predicted  and ground truth mesh vertices~\cite{zimmermann2019freihand};  and the
$F$-score~\cite{zimmermann2019freihand},  defined as  the harmonic  mean between
recall and  precision between two meshes  given a distance threshold.   The mesh
error  and \textit{F}-score  are obtained  after aligning  the predicted  meshes
using  Procrustes alignment  with the  ground truth  meshes and  hence does  not
measure the accuracy of wrist joint rotation.  The mean joint error on the other
hand considers wrist joint location  as the 3D points  are not rotated
before evaluation.

To understand the effect of joint direction predictions on the overall accuracy,
we evaluate  the results  of the  MANO fitting  by dropping  the second  term in
Eq.~\eqref{eq:manoFitEst}.  We also compare our  results with the hand branch of
\cite{hasson2019learning}, a  very recent work  that predicts the MANO  pose and
shape parameters directly from a single RGB image, retrained on our dataset.  As
shown in Table~\ref{tab:tablePoseEst}, predicting joint directions along with 2D
joint locations  significantly improves the  hand pose estimation  accuracy.  It
can also be inferred that predicting 2D hand joint locations and fitting MANO model to them is  more   accurate  than   direct   MANO  parameter   predictions  as   in
\cite{hasson2019learning}.      Qualitative     results     are     shown     in
Fig.~\ref{fig:baseline_qualitative}. The last row shows that our method robustly
predicts hand poses even when interacting with unknown objects.

\comment{
  \begin{figure}
    \begin{center}

      \includegraphics[page=17, width=0.3\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes}
      \includegraphics[page=9, width=0.3\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes}
      \includegraphics[page=6, width=0.3\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes} \\
      \includegraphics[page=10, width=0.3\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes}
      \includegraphics[page=19, width=0.3\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes}
      \includegraphics[page=7, width=0.3\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes} \\
      \includegraphics[page=18, width=0.3\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes}
      \includegraphics[page=20, width=0.3\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes}
      \includegraphics[page=12, width=0.3\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes}
    \end{center}
    \vspace{-2mm}
    \caption{Qualitative results for single RGB frame based hand pose estimation method. We recover hand poses when it is heavily occluded by objects and in cluttered scenes.}
    \label{fig:baseline_qualitative}
    \vspace{-5mm}
  \end{figure}
}

\begin{figure}
  \begin{center}

    \includegraphics[page=17, width=0.24\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes2}
    \includegraphics[page=9, width=0.24\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes2}
    \includegraphics[page=6, width=0.24\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes2}
    \includegraphics[page=10, width=0.24\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes2}\\
    \includegraphics[page=19, width=0.24\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes2}
    \includegraphics[page=7, width=0.24\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes2}
    \includegraphics[page=18, width=0.24\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes2}
    \includegraphics[page=20, width=0.24\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes2}\\
    \includegraphics[page=22, width=0.24\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes2}
    \includegraphics[page=24, width=0.24\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes2}
    \includegraphics[page=25, width=0.24\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes2}
    \includegraphics[page=28, width=0.24\linewidth, trim=6cm 1.35cm 6cm 1.9cm,clip]{figures/singleFramePose/handposes2}
  \end{center}
  \vspace{-2mm}
  \caption{Qualitative results of  our single color image  hand pose estimation
    method. It can recover hand poses even when the hand is heavily occluded by objects and in
    cluttered scenes. The last row shows it can handle unseen objects.}
  \label{fig:baseline_qualitative}
  \vspace{-5mm}
\end{figure}

\comment{
  \paragraph{PCK metric: Percentage of Correct Keypoints in a Side View.}
  We use  a second  RGB camera registered  with respect  to the  RGB-D camera  used to
  capture  the sequences.   This camera  has a  viewpoint that  is approximately
  orthogonal to  the RGB-D  camera.  We use  the first camera  to obtain  the pose
  annotations and the second camera for  validation only. We manually annotate the
  hand pose  with the 2D  joint locations of visible  joints in the  second camera
  view for 10 randomly chosen frames  of 15 different sequences, totaling over 150
  frames.   Further, we  annotated  the object  pose by  providing  the 2D  corner
  locations of visible selected points in the second camera view totaling over 100
  frames.

  To evaluate the PCK metric, we project  the estimated 3D pose annotations on the
  second camera, and  compute the 2D distances between the  manual annotations and
  the projected automatic  annotations. Fig.~\ref{fig:anno_frameswithin} shows the
  PCK metric  for varying  distance thresholds, \ie, the  percentage of  3D points
  that have an error smaller than  a threshold. The final stage of the global optimization discussed in Section \ref{GO} increases the accuracy of annotations by about 7\% by refining the poses. Qualitative results before and after the final stage of global optimization are shown in Fig.~\ref{fig:global_qualitative}. More qualitative examples on different objects, persons and poses are shown in Fig.~\ref{fig:anno_qualitative}.

  \begin{figure}
    \begin{center}
      \includegraphics[width=0.45\linewidth]{plots/PCK_Dataset_Hand}
      \includegraphics[width=0.45\linewidth]{plots/PCK_Dataset_Object}
    \end{center}
    \vspace{-5mm}
    \caption{PCK metric (before and after final refinement stage) for evaluating the proposed pose annotation method on 15 sequences with  varying distance thresholds.  Left: Hand.  Right: Object.}
    \label{fig:anno_frameswithin}
  \end{figure}

  \begin{figure}
    \begin{center}
      \includegraphics[width=0.28\linewidth]{figures/mask_0045pngimg2_SM1_b.png}
      \includegraphics[width=0.28\linewidth]{figures/mask_0304pngimg2_MC1_b.png}
      \includegraphics[width=0.28\linewidth]{figures/mask_0652jpgimg2_SM2_b.jpg} \\
      \includegraphics[width=0.28\linewidth]{figures/mask_0045jpgimg2_SM1_a.jpg}
      \includegraphics[width=0.28\linewidth]{figures/mask_0304pngimg2_MC1_a.jpg}
      \includegraphics[width=0.28\linewidth]{figures/mask_0652jpgimg2_SM2_a.jpg}
    \end{center}
    \vspace{-2mm}
    \caption{Qualitative results of the proposed annotation
      method. Each column shows the 3D models of the hand and
      object superimposed on the images from the side view camera
      before and after refinement stage, respectively. The final
      refinement still improves the estimated poses.}
    \vspace{-6mm}
    \label{fig:global_qualitative}
  \end{figure}

  \begin{figure}
    \hspace{-2mm}
    \begin{center}
      \includegraphics[width=0.6\linewidth]{plots/Dataset_depth_errors}
    \end{center}
    \vspace{-4mm}
    \caption{Reconstruction error. We plot the fraction of depth residuals where the depth residual is smaller than a threshold. A large fraction of errors are well below 10~mm, which shows that the articulated models can explain the captured data accurately.}
    \label{fig:depth_errors}
  \end{figure}

  \begin{figure}
    \begin{center}
      \includegraphics[width=0.65\linewidth]{plots/PCK_HONet_hand}
    \end{center}
    \vspace{-5mm}
    \caption{Evaluation of our baseline for hand pose estimation. We plot the percentage of correct keypoints~(PCK) over distance threshold. The dashed curve shows that the synthetic images used to train the image segmentation are clearly not sufficient to train this baseline, even if they work well for the segmentation task. }
    \label{fig:baseline_hand}
    \vspace{-5mm}
  \end{figure}

  \vspace{-6mm}

  \paragraph{Reconstruction error.}
  As an additional  evaluation, we rely on  a reconstruction error.  To  do so, we
  compute the  difference between the  captured depth maps  and the the  depth maps
  generated  using the  rendered  models of  the  hand and  the  object under  the
  estimated  poses.    Fig.~\ref{fig:depth_errors}  shows  the   evaluation for three different objects \textit{Mustard bottle, Cracker box} and \textit{Sugar box}.   The
  majority of all  errors, around 90\%, are below 10~mm.  The remaining errors can
  be attributed to  depth noise from the  sensor and the wrist  and forearm, which
  are not present in our model.
}

\subsection{3D Hand and Object Poses}
\label{sec:poses}

We    aim   to    estimate    the   3D    poses    $\mathcal{P}   =    \{(\text{p}_\text{h}^t,
\text{p}_\text{o}^t)\}^{N_F}_{t=1}$ for both  the hand and the object in  all the images
of the sequence.  We adopt the MANO hand model~\cite{romero2017embodied} and use
the objects from the YCB-Video dataset~\cite{posecnn2018} as their corresponding
3D models are available and of good quality. The MANO hand pose $\text{p}_\text{h}^t \in
\mathbb{R}^{51}$ consists of 45 DoF~(3 DoF for each of the 15 finger joints) plus 6 DoF
for rotation  and translation of the  wrist joint.  The 15  joints together with
the wrist joint form  a kinematic tree with  the wrist joint node as  the first parent
node.  In  addition to  the pose  parameters $\text{p}_\text{h}^t$,  the hand  model has
shape parameters $\boldsymbol{\beta} \in \mathbb{R}^{10}$ that are  fixed for a given person and we
follow a  method similar to  \cite{Tan_2016_CVPR} to estimate  these parameters.
More  details  about  the  shape   parameter  estimation  are  provided  in  the
supplementary material. The object  pose $\text{p}_\text{o}^t \in \text{SE}(3)$ consists
of 6 DoF for global rotation and translation.

\subsection{Cost Function}
\label{sec:cost}

We  formulate   the  hand+object  pose   estimation  as  an   energy
minimization problem:
\begin{equation}
\label{eq:global}
\hat{\mathcal{P}} = \arg\min\limits_{\mathcal{P}} \sum^{N_F}_{t=1}
\big(E_\mathcal{D}(\text{p}_\text{h}^t, \text{p}_\text{o}^t) + E_\mathcal{C}(\text{p}_\text{h}^t, \text{p}_\text{o}^t)\big)\> ,
\end{equation}
where $E_\mathcal{D}$ and  $E_\mathcal{C}$ represent the energy  from data terms
and constraints, respectively.  We define $E_\mathcal{D}$ as,
\begin{equation}
  \begin{array}{r}
    \!\!\!\!\mvleft
    E_\mathcal{D}(\text{p}_\text{h}^t, \text{p}_\text{o}^t) = \sum\limits^{N_C}_{c=1} \Big( \alpha E_\text{mask}(\text{I}^t_c,\text{p}_\text{h}^t,\text{p}_\text{o}^t) \:+ \beta E_\text{dpt}(\text{D}^t_c, \text{p}_\text{h}^t,\text{p}_\text{o}^t)  + \\[0.5em]
   \!\!\!\! \gamma E_\text{j2D}(\text{I}^t_c, \text{p}_\text{h}^t) \Big) \> + \delta E_\text{3D}(\{\text{D}^t_c\}_{c=1..N_C}, \text{p}_\text{h}^t,\text{p}_\text{o}^t) \> ,
\label{eq:dataTerm}
\end{array}
\end{equation}

\noindent where $E_\text{mask}(\cdot)$ is a  silhouette discrepancy term, $E_\text{dpt}(\cdot)$ a
depth  residual  term,  $E_\text{j2D}(\cdot)$  a 2D  error  in  hand  joint
locations,  and $E_\text{3D}(\cdot)$  a  3D  distance term.  This  last  term is  not
absolutely necessary,  however, we  observed  that it  significantly  speeds up  convergence.
$\alpha$, $\beta$,  $\gamma$, $\delta$ are weights.

The constraints energy $E_\mathcal{C}$ is defined as,
\begin{align}
E_\mathcal{C}(\text{p}_\text{h}^t, \text{p}_\text{o}^t)  = \; & \epsilon E_\text{joint}(\text{p}_\text{h}^t) \; + \zeta E_\text{phy}(\text{p}_\text{h}^t,\text{p}_\text{o}^t) \; + \nonumber\\
& \eta E_\text{tc}(\text{p}_\text{h}^t,\text{p}_\text{o}^t,\text{p}_\text{h}^{t-1},\text{p}_\text{o}^{t-1},\text{p}_\text{h}^{t-2},\text{p}_\text{o}^{t-2}) \> ,
\end{align}
where $E_\text{joint}(\cdot)$  denotes a prior on  the hand pose to  prevent unnatural
poses, $E_\text{phy}(\cdot)$ is a physical plausibility term ensuring the hand and the
object  do not  interpenetrate,  and $E_\text{tc}(\cdot)$  is  a temporal  consistency
term. The  terms are weighted by  parameters $\epsilon$, $\zeta$ and  $\eta$.

We detail  each of the terms  in $E_\mathcal{D}$ and $E_\mathcal{C}$  below. For
simplicity, we  omit the  frame index  $t$ from our  above notation  except when
necessary.

\vspace{-0.3cm}
\paragraph{Silhouette discrepancy term $E_\text{mask}$. }
The $E_\text{mask}(\cdot)$ term compares the silhouettes  of the hand and the object models
rendered  with the  current estimated  poses and  their segmentation  masks.  We
obtain a segmentation $S_c(\text{I})$ of the hand and the object in the color image $\text{I}$ of camera $c$ using
DeepLabv3~\cite{deeplabv2017}  trained   on  images  created   by  synthetically
over-laying  and under-laying  images of  hands on  YCB objects.  More
  details about this step are given in the supplementary material. The hand and
object  models  are  rendered  on   the  camera  plane  using  a  differentiable
renderer~\cite{Henderson2019},  which  enables   computing  the  derivatives  of
$E_\text{mask}$ with  respect to the pose  parameters. The silhouette of  the hand and
object rendered on  camera $c$ is denoted by $\text{RS}_c(\text{p}_\text{h},\text{p}_\text{o})$  and the silhouette
discrepancy is defined as,
\begin{align}
E_\text{mask}(\text{I}_c, \text{p}_\text{h}, \text{p}_\text{o}) = \lVert \text{RS}_c(\text{p}_\text{h}, \text{p}_\text{o}) - S(\text{I}_c) \rVert^2 \> .
\end{align}

\paragraph{Depth residual term $E_\text{dpt}$. }
The depth residual term is similar to the segmentation discrepancy term:
\begin{equation}
E_\text{dpt}(\text{D}_c, \text{p}_\text{h}, \text{p}_\text{o}) = \text{Tukey}(\lVert \text{RD}_c(\text{p}_\text{h}, \text{p}_\text{o}) - \text{D}_c \rVert) \> ,
\end{equation}
where $\text{RD}_c(\text{p}_\text{h}, \text{p}_\text{o})$ is the depth rendering of the hand and
the object under their current estimated poses $\text{p}_\text{h}$ and $\text{p}_\text{o}$.  The
Tukey function is a robust estimator that  is similar to the $\ell_2$ loss close
to  0, and  constant after  a threshold.   It is  useful to  be robust  to small
deviations in the scale  and shape of the hand and object  models and also noise
in the  captured depth maps.   $E_\text{dpt}$ is differentiable as we  employ a
differentiable renderer~\cite{Henderson2019} for rendering the depth maps.

\vspace{-0.3cm}
\paragraph{2D Joint error term $E_\text{j2D}$. }

The 2D joint error term is defined as,
\begin{equation}
E_\text{j2D}(\text{I}_c, \text{p}_\text{h}) = \sum_{i=1}^{21} \text{h}[i] \Big\lVert
\text{proj}_c(\text{J}_{\text{p}_\text{h}}[i]) - \text{K}_c[i] \Big\rVert^2 \> ,
\label{eq:j2dErr}
\end{equation}
where $\text{J}_{\text{p}_\text{h}}[i]$ denotes the $i^\text{th}$ 3D hand joint location under pose $\text{p}_\text{h}$, the $\text{proj}_c(\cdot)$ operator projects it onto camera $c$, $\text{K}_c[i]$ is its predicted 2D location, and $\text{h}[i]$ its confidence.  The 21 hand joints in $E_\text{j2D}(\cdot)$ consist of 15 finger joints, 5 finger tips, and the wrist joint.

In practice,  we take  the $\text{K}_c[i]$ as  the locations of  the maximum  values of
heatmaps, and  the $\text{h}[i]$  as the  maximum values  themselves. To  predict these
heatmaps, we trained a CNN based  on the architecture of \cite{Wei16}.  Training
data  come   from  an  initial   dataset~\cite{ho3d_old}  we  created   using  a
semi-automatic method. This  dataset is made of 15,000 frames  from 15 sequences
in a single camera setup. We manually initialized the grasp pose and object pose
for the first  frame of each sequence.  The manipulators  were asked to maintain
their grasp poses as rigid as possible to make the registration easier.  We then
ran the optimization  stages for the single camera case  described in Section~\ref{sec:singleCameraSetup}.  After
optimization, we  augmented the  resulting dataset by  scaling and  rotating the
images,     and      adding     images     from     the      Panoptic     Studio
dataset~\cite{xiang2019monocular}, which contain 3D annotations for hands.

\vspace{-0.3cm}
\paragraph{3D error term $E_\text{3D}$. }
This term  is not  absolutely necessary as  the depth  information from  all the  cameras is
already exploited by $E_\text{dpt}$, however it accelerates the convergence by guiding
the optimization towards the  minimum even from far away.  We  build a point cloud
$\text{P}$ by  merging the depth maps  from the RGB-D cameras  after transforming
them  to   a  common  reference  frame.    More  details  on  the   point  cloud
reconstruction can be found in the supplementary material.

We segment  $\text{P}$ into  an object  point cloud $\text{P}_\oo$  and a  hand point
cloud $\text{P}_\hh$ using  the segmentation mask $S_c(\text{I})$ in  each camera image.
At  each iteration  of the  optimization, for  each point  $\text{P}_\oo[j]$ of  the
object point cloud, we look for the closest vertex $\text{V}_\text{o}[j^*]$ on the object
mesh, and for each point $\text{P}_\hh[k]$ of  the hand point cloud, we look for the
closest vertex  $\text{V}_\text{h}[k^*]$ on  the hand mesh.   $E_\text{3D}(\text{P}, \text{p}_\text{h},
\text{p}_\text{o})$ is then defined as,
\begin{align}
\sum_j &\big\lVert \text{P}_\oo[j] - \text{V}_\text{o}[j^*]\big\rVert^2 + \sum_k \big\lVert \text{P}_\hh[k] - \text{V}_\text{h}[k^*]\big\rVert^2 \> .
\end{align}

\vspace{-0.5cm}
\paragraph{Joint angle constraint $E_\text{joint}$. }
This  term imposes  restrictions on  the 15  joints of  the hand  to ensure  the
resulting  pose is  natural.   The  three-dimensional  rotation  of  a joint  is
parameterized using the axis-angle representation in MANO model, resulting in 45
joint angle  parameters.  A common  solution when using MANO model is to optimize in
the PCA space of 3D joint angles with an $\ell_2$ regularizer~\cite{zimmermann2019freihand, boukhayma20193d} for pose coefficients. However, we  observed  in practice  that
optimizing Eq.~\ref{eq:global} in PCA space had less expressibility: some of the complex grasp poses in our dataset could not be accurately expressed in the PCA space, which was constructed with relatively simpler grasp poses and free-hand gestures.
  Instead, we optimize on joint angles directly and derive our
  own limits  for each  of the 45  joint parameters~(please  refer to supplementary
  material for these limits).  As in \cite{Zhou16c}, the joint angle constraint
term $E_\text{joint}(\text{p}_\text{h}^t)$ is given by,
\begin{equation}
\sum_{i=1}^{45} \max(\underline{a_i} - a[i], 0) + \max(a[i] -
\overline{a_i}, 0) \> ,
\label{eq:ejoint}
\end{equation}
where $a[i]$  denotes the $i^{th}$ joint  angle parameter for pose  $\text{p}_\text{h}$, and
$\underline{a_i}$ and $\overline{a_i}$ correspond to its lower and upper limits.

\begin{figure*}[ht]
  \begin{center}
    \includegraphics[width=0.85\linewidth, trim=.2cm 3.5cm .0cm 2cm,clip]{figures/singleview_pipeline}
  \end{center}
\vspace{-4mm}
\caption{The different stages of the multi-camera and single camera setups. See Section \ref{sec:fullOpt} for more details.}
\label{fig:methodOverview}
\vspace{-10px}
\end{figure*}

\vspace{-0.3cm}
\paragraph{Physical plausibility term $E_\text{phy}$. }
During optimization, the  hand model might penetrate the object  model, which is
physically not possible.  To avoid this, we add a repulsion term that pushes the
object and  the hand  apart if  they interpenetrate each  other.  For  each hand
vertex $V_\text{h}[m]$, the amount of penetration $\Gamma[m]$ is taken as,
\begin{equation}
\Gamma[m] = \max(-\mathbf{n}_\text{o}\big(\text{V}_\text{o}[m^*]\big)^T\big(\text{V}_\text{h}[m] -\text{V}_\text{o}[m^*]\big), 0) \> ,
\end{equation}
where  $\text{V}_\text{o}[m^*]$  is   the  vertex  on  object  closest   to  hand  vertex
$\text{V}_\text{h}[m]$, and the $\mathbf{n}_\text{o}(\cdot)$ operator  provides the normal vector for
a vertex.   In words, the amount  of penetration is estimated  by projecting the
vector  joining a  hand vertex  and its  nearest object  vertex onto  the normal
vector at  the object vertex location.   The physical plausibility term  is then
defined as,
\begin{equation}
E_\text{phy}(\text{p}^t_\text{h},\text{p}^t_\text{o}) = \sum_m \exp{\big(w~\Gamma[m]\big)} \> .
\end{equation}
We use  $w = 5$ in  practice, and only a subsampled  set of vertices of  the hand to
compute $E_\text{phy}$ efficiently.

\paragraph{Temporal consistency term $E_\text{tc}$.}

The previous  terms are all applied  to each frame independently.   The temporal
consistency term $E_\text{tc}$  allows us to constrain together the  poses for all the
frames. We apply a 0-th and 1-st order  motion model on both the hand and object
poses:
\begin{equation}
\begin{array}{l}
E_\text{tc}(\text{p}_\text{h}^t,\text{p}_\text{o}^t,\text{p}_\text{h}^{t-1},\text{p}_\text{o}^{t-1},\text{p}_\text{h}^{t-2},\text{p}_\text{o}^{t-2}) = \\[0.5em] \nonumber
\lVert \Delta_h^t \rVert^2 + \lVert \Delta_o^t \rVert^2 +
\lVert \Delta_h^t - \Delta_h^{t-1} \rVert^2 + \lVert \Delta_o^{t} -
\Delta_o^{t-1} \rVert^2 \> ,
\end{array}
\end{equation}
where $\Delta_h^t = \text{p}_\text{h}^t - \text{p}_\text{h}^{t-1}$ and $\Delta_o^t = \text{p}_\text{o}^t
- \text{p}_\text{o}^{t-1}$.  Since we  optimize a sum of these terms  over the sequence,
this effectively constrains all the poses together.

\section{Introduction}

\input{intro.tex}

\section{Related Work}

\input{rel_work.tex}

\section{3D Annotation Method}
\label{sec:method}

\input{method.tex}

\section{Optimization}
\label{sec:fullOpt}

\input{optimization.tex}

\section{Monocular RGB based 3D Hand Pose
\label{sec:handposeSingleFrameSec}

\input{method2.tex}

\section{Benchmarking \datasetname}

\input{eval.tex}

\section{Conclusion}

We introduced a fully automatic method to annotate images of a hand manipulating
an  object with  their  3D poses,  even under  large  occlusions, by  exploiting
temporal  consistency. We also introduced the first  markerless  dataset of  color images  for
benchmarking 3D  hand+object pose estimation.  To demonstrate the  usefulness of
our dataset, we  proposed a method  for predicting the 3D pose  of the hand
from  a single  color  image. Another  future  application is  the  joint estimation  of
hand+object poses from a single RGB frame.

The lack  of high quality  segmentations (we had to  use a synthetic  dataset as
explained in  Section~\ref{sec:method}) of the hand  sometimes affects accuracy.
Improving  these segmentations  and/or introducing  a attraction  term and  physics
constraints as  in~\cite{RealtimeHO_ECCV2016, Tzionas2016} would  further
improve our annotations.

\section{Acknowledgement}
This work  was supported  by the  Christian Doppler  Laboratory for  Semantic 3D
Computer Vision, funded in part by Qualcomm Inc.

\title{HOnnotate: A method for 3D Annotation of Hand and Object Poses \\ Supplementary Material}
\maketitle
\setcounter{section}{0}
\input{supplementarySec.tex}

{\small
\bibliographystyle{ieee_fullname}

\subsection{2D Joint Detection}

\subsection{Multi-Camera Setup}
\label{sec:multiCameraSetup}

\paragraph{Initialization.}
In the multi-camera setup, we obtain a first estimate $\tilde{\text{p}}_\text{h:w}^0$ for the
hand pose in the first frame $(t=0)$ as,
\begin{equation}
\label{eq:multiViewInit}
\tilde{\text{p}}_\text{h:w}^0   =  \arg\min\limits_{\text{p}_\text{h:w}}   \sum_{c=1}^{N_C}
E_\text{j2D}(\text{I}_c^0, \text{p}_\text{h:w}) + \nu E_\text{joint}(\text{p}_\text{h:w}) \> .
\end{equation}
We use the Dogleg  optimizer~\cite{ConnGoulToin00} to perform this optimization.
A  first estimate  $\tilde{\text{p}}_\text{o}^0$  for  the object  pose  in  this frame  is
obtained using  \cite{Rad17} trained by  synthetically over-laying hands  on YCB
objects as explained in Section~\ref{sec:cost}.

\paragraph{Single-frame joint pose optimization.}
We then obtain  estimates $\tilde{\text{p}}_\text{h:w}^t$ and $\tilde{\text{p}}_\text{o}^t$  for all the
other  frames ($t = 1..N_F$)  by   tracking.   We   minimize  $\big(
E_\mathcal{D}(\text{p}_\text{h:w}^t,  \text{p}_\text{o}^t)  + E_\mathcal{C}(\text{p}_\text{h:w}^t,  \text{p}_\text{o}^t)  \big)$
w.r.t.   $\text{p}_\text{h:w}^t$  and   $\text{p}_\text{o}^t$,  using  $\tilde{\text{p}}_\text{h:w}^{t-1}$  and
$\tilde{\text{p}}_\text{o}^{t-1}$ for initialization.

\paragraph{Multi-frame joint pose optimization.}
We finally  perform a  full optimization of  Eq.~\eqref{eq:global} w.r.t. $\text{p}_\text{h}^t$ and
$\text{p}_\text{o}^t$ for $t=0..N_F$ over  all the
frames simultaneously using  estimates $\tilde{\text{p}}_\text{h:w}^t$ and $\tilde{\text{p}}_\text{o}^t$
for    initialization.     Due    to    memory    constraints,    we    optimize
Eq.~\eqref{eq:global}  in  batches instead  of  considering  all the  frames  in
sequence.   We use  a  batch size  of 20  frames  with $\alpha=20$,  $\beta=20$,
$\gamma=5\times10^{-5}$,    $\delta=50$,    $\epsilon=100$,   $\zeta=50$,    and
$\eta=100$,  and  the  Adam  optimizer  with  learning  rate  of  0.01  for  100
iterations.

\subsection{Single-Camera Setup}
\label{sec:singleCameraSetup}

\paragraph{Initialization.}
In the single camera  setup, as we assume that the  grasp pose varies marginally
across the sequence,  we initially make the assumption that  it remains constant
throughout the sequence.  In order to account for the minor changes which occur
in practice, we relax this assumption  in the latter stages of the optimization.
We thus obtain initial estimates $\tilde{\text{p}}_\text{h:w}^t$ for the hand poses as,
\begin{align}
\label{eq:singleViewPoseEst}
\{ \tilde{\text{p}}_\text{h:w}^t\}_t = \arg\min_{\{\text{p}_\text{h:w}^t\}_t} \sum_t E_\text{j2D}(\text{I}^t, \text{p}_\text{h:w}^t) + \nu E_\text{joint}(\text{p}_\text{h:w}^t) \> ,
\end{align}
where the  joint angle parameters  are constrained to be  the same over  all the
frames, and only the rotation and translation parameters for the wrist joint can
be different.   In practice,  we perform  this optimization  only over  a random
subset $\Omega$ of the frames to save time.  We set $\nu = 50$, size of $\Omega$
to  20  and use  the  Dogleg  optimizer~\cite{ConnGoulToin00}.  First  estimates
$\tilde{\text{p}}_\text{o}^t$ for  the object  poses in  $\Omega$ are  obtained as  for the
multi-camera setup.

From the $\tilde{\text{p}}_\text{h:w}^t$ and the $\tilde{\text{p}}_\text{o}^t$, we can compute the grasp
pose in the  \emph{object coordinate system} $\tilde{\text{p}}_\text{h:o}^t$,  which is assumed
to be constant at this stage. The initial estimate of the constant grasp pose $\tilde{\text{p}}_\text{h:o}$  is taken as the average of $\{\tilde{\text{p}}_\text{h:o}^t\}_{t\in\Omega}$

\vspace{-0.5px}
\paragraph{Grasp pose estimation.}
We obtain  a better estimate of  the grasp pose $\hat{\text{p}}_\text{h:o}$  and object poses
$\hat{\text{p}}_\text{o}^t$ under fixed grasp pose assumption as,
\begin{equation}
\begin{array}{r}
\hat{\text{p}}_\text{h:o}, \{\hat{\text{p}}_\text{o}^t\}_{t \in \Omega} = \arg\min_{\text{p}_\text{h:o},\{\text{p}_\text{o}^t\}_{t \in \Omega}} \sum_{t \in \Omega} E_\mathcal{D}(f_{\text{p}_\text{o}^t}(\text{p}_\text{h:o}), \text{p}_\text{o}^t) + \\[0.5em]
\zeta E_\text{phy}(f_{\text{p}_\text{o}^t}(\text{p}_\text{h:o}), \text{p}_\text{o}^t) + \epsilon E_\text{joint}(\text{p}_\text{h:o}), \\ [0.5em]
\end{array}
\end{equation}
wrt $\text{p}_\text{h:o}$ and $\text{p}_\text{o}^t$ over the frames  in $\Omega$, using $\tilde{\text{p}}_\text{h:o}$  and $\tilde{\text{p}}_\text{o}^t$
for  initialization. $f_{\text{p}_\text{o}^t}(\cdot)$ converts the grasp pose to the hand pose in the world coordinate system given object pose $\text{p}_\text{o}^t$.  This  optimization  accounts for  the  mutual  occlusions between hand and  object.
\vspace{-0.4cm}
\paragraph{Object pose estimation.}
Having a good estimate $\hat{\text{p}}_\text{h:o}$ for  the grasp pose, we obtain object pose
estimates    $\hat{\text{p}}_\text{o}^t$    for    all    the    frames    by    minimizing
$E_\mathcal{D}\big(f_{\text{p}_\text{o}^t}(\hat{\text{p}}_\text{h:o}),\text{p}_\text{o}^t\big)$ wrt $\text{p}_\text{o}^t$   over  each   frame  independently. We  use
$\hat{\text{p}}_\text{o}^{t-1}$  to  initialize  the  optimization  over  $\text{p}_\text{o}^t$ at frame $t$,
except for $\hat{\text{p}}_\text{o}^0$ where $\tilde{\text{p}}_\text{o}^0$  is used.  Note that the hand
pose is not optimized in this stage.

\vspace{-0.3cm}
\paragraph{Multi-frame joint hand+object pose refinement.}
In this  final stage, we  allow variations in the  grasp pose across  frames and
introduce temporal constraints.  We thus optimize Eq.~\eqref{eq:global} w.r.t. $\{(\text{p}_\text{h}^t,
\text{p}_\text{o}^t)\}^{N_F}_{t=1}$ over  all the frames  simultaneously, using  pose parameters
  $\hat{\text{p}}_\text{o}^t$, $\hat{\text{p}}_\text{h}^t=f_{\hat{\text{p}}_\text{o}^t}(\hat{\text{p}}_\text{h:o})$ estimated  in  the  previous   stages  as initialization for $\text{p}_\text{o}^t$ and $\text{p}_\text{h}^t$.

\subsection{3D Object Pose Estimation}

Estimating the  3D pose of  an object from  a single frame  is still one  of the
fundamental problems of Computer Vision. Some  methods are now robust to partial
occlusions~\cite{Oberweger2018,peng2019pvnet,hu2019segpose}, but many works rely on RGB-D data to
handle this problem~\cite{Buch2017,Mitash2017,Zhang2017,Kehl16},  by fitting the
3D object model to depth data. This  can fail when a hand grasps the object,
since the surface of the hand can be mistaken for the surface of the object.

\subsection{3D Hand Pose Estimation}

Single image hand pose estimation is also a very popular problem in Computer Vision, and approaches can be divided into discriminative and generative methods. Discriminative approaches directly predict the joint locations from RGB or RGB-D images.  Recent works based on Deep Networks \cite{Tompson14b,Xu16,Neverova17,Oberweger17,mueller2017real,Zimmermann2017,ge2018hand} show remarkable performance, compared to early works based on Random Forests such as \cite{keskin2012hand}.  However, discriminative methods perform poorly in case of partial occlusion.

Generative approaches take advantage of a hand model and its kinematic structure to generate hand pose hypotheses that are physically plausible~\cite{sridhar2013interactive,Qian2014realtime,sharp2015accurate,Tzionas2014capturing,melax2013dynamics,de2011model,ye2016spatial}.  \cite{Mueller2018,Panteleris2018} predict 2D joint locations and then lift them to 3D.  Generative approaches are usually accurate and can be made robust to partial occlusions.  They typically rely on some pose prior, which may require manual initialization or result in drift when tracking.

Our work is  related to both discriminative and generative  approaches: we use a
generative approach within a global  optimization framework to generate the pose
annotations,  and  use  a  discriminative  method  to  initialize  this  complex
optimization.  \cite{Wan_2019_CVPR}  also combines generative  and discriminative
approaches to train  a network in a self-supervised setting.   However, they only
consider hands.   We also train  a discriminative  method using our  dataset, to
predict the hand poses which are robust to occlusions from interacting objects.

\subsection{Synthetic Images for 3D Pose Estimation}

Being able to train discriminative methods on synthetic data is valuable as it is difficult to acquire annotations for real images~\cite{Zimmermann2017}.  \cite{hasson2019learning,Rad2018b} show that because of the domain gap between synthetic and real images, training on synthetic images only results in sub-optimal performance.  A GAN method was used in \cite{Mueller2018} to make synthetic images of hands more realistic.  While using synthetic images remains appealing for many problems, creating the virtual scenes can be expensive and time-consuming.  Generating animated realistic hand grasps of various objects, as it would be required to solve the problem considered in this paper remains challenging. Being able to use real sequences for training has thus also its advantages. Moreover, evaluation should be performed on real images.

\subsection{Joint Hand+Object Pose Estimation}

Early         approaches        for         joint        hand+object         pose
estimation~\cite{Oikonomidis2011full,Wang11d,Ballan2012motion}  typically relied
on  multi-view camera  setups, and  frame-by-frame tracking  methods, which  may
require      careful      initialization      and     drift      over      time.
\cite{panteleris20153d,Tzionas_2015_ICCV}  propose generative  methods to  track
finger   contact    points   for   in-hand   RGB-D    object   shape   scanning.
\cite{pham2015capturing,Pham_2015_CVPR} consider sensing from vision to estimate
contact forces during hand+object interactions  using a single RGB-D camera, and
then estimate the hand and the  object pose.  However, these methods are limited
to small occlusions.

\cite{kyriazis2013physically,Tzionas2016} propose to use a physics simulator and
a  3D renderer  for  frame-to-frame tracking  of hands  and  objects from  RGB-D.
\cite{Kyriazis14} uses  an ensemble  of Collaborative Trackers  for multi-object
and multiple  hand tracking from  RGB-D images.   The accuracy of  these methods
seems to  be qualitatively  high, but  as the ground  truth acquisition in
a real-world is known to be hard, they evaluate the proposed method on
synthetic datasets, or by measuring the  standard deviation of the difference in
hand/object poses during a grasping scenario.

\cite{Tsoli_2018_ECCV} considers the problem of  tracking a deformable object in
interaction with a hand, by optimizing  an energy function on the appearance and
the kinematics  of the hand,  together with hand+object  contact configurations.
However, it is  evaluated quantitatively only on synthetic  images, which points
to the difficulty  of evaluation on real data.  In  addition, they only consider
scenarios where the  hand is visible from  a top view, restricting  the range of
the hand poses and not allowing occlusions.

Very recently,  \cite{kokic2019learning} uses a  coarse hand pose  estimation to
retrieve the 3D pose and shape of hand-held objects. However, they only consider
a   specific  type   of   object  and   do  not   estimate   the  object   pose.
\cite{hasson2019learning}  presents a  model  with contact  loss that  considers
physically feasible hand+object interaction  to improve grasp quality.  However,
to  estimate 3D  hand pose,  they  predict PCA  components for  the pose,  which
results  in  lower   accuracy  compared  to  ours,   as  our  experiments
show. \cite{tekin2019h}  proposes a deep model  to jointly predict 3D  hand and
object poses from egocentric views, but the absence of physical constraints might
result in infeasible grasps.

\subsection{Hand+Object Datasets}

Several datasets for  hand+object interactions have already  been proposed. Many
works    provide    egocentric   RGB    or    RGB-D    sequences   for    action
recognition~\cite{bullock2015yale,cai2015scalable,fathi2011learning,bambach2015lending,rogez2015understanding}\comment{,luo2017scene,Tzionas_2015_ICCV}.
However,
they  focus  on   grasp  and  action  labels  and  do   not  provide  3D  poses.
\cite{choi2017robust,rogez20143d,mueller2017real,Tsoli_2018_ECCV}
generate synthetic datasets with 3D hand pose  annotations, but fine interaction between a
hand and an object remains difficult to generate accurately.

\cite{Tzionas2016,Tzionas2014capturing}  captured sequences  in  the context  of
hand+hand  and   hand+object  interaction,   with  2D  hand   annotations  only.
\cite{Myanganbayar18} collected  a dataset of  real RGB images of  hands holding
objects.  They  also provide 2D joint  annotations of pairs of  non-occluded and
occluded hands,  by removing  the object  from the grasp  of the  subject, while
maintaining  their hand  in  the same  pose.   \cite{goudie20173d} proposes  two
datasets, a hand+object  segmentation dataset, and a  hand+object pose estimation
dataset. However,  for both  datasets, the  background pixels  have been  set to
zero, and the training  images only consist of a hand  interacting with a tennis
ball.   They provide  hand  pose  annotations and  object  positions, by  manually
labelling the joints and using a generative method to refine the joint positions.
\cite{JooSS18} generate a large scale dataset with full body pose and hand pose annotations in a multi-view setup. They use a generative approach to fit the body and hand models to 3D keypoints and point cloud. However, their dataset focuses on total body pose annotation and not hand+object interactions exclusively and does not provide object pose annotations.

\cite{RealtimeHO_ECCV2016} proposed  an RGB-D dataset  of a hand  manipulating a
cube, which  contains manual ground  truth for  both fingertip positions  and 3D
poses of the  cube.  \cite{pham2018hand} collected a dataset  where they measure
motion and force under different  object-grasp configurations using sensors, but
do   not   provide  3D   poses.    In   contrast   to  these   previous   works,
\cite{garcia2018first} provides a dataset of hand and object interactions with 3D annotations
for both hand joints and object pose.  They used a motion capture system made of
magnetic sensors  attached to  the user's hand and  to the  object in  order to
obtain hand 3D pose annotations in  RGB-D video sequences.  However, this changes
the appearance of the hand in the color images as the sensors and the tape attaching
them are visible.

Very recently,  \cite{hasson2019learning} introduced  ObMan, a large  dataset of
images of  hands grasping  objects. The images in ObMan dataset are synthetic  and the
grasps  are generated  using an  algorithm from  robotics.  Even  more recently,
\cite{zimmermann2019freihand} proposed a multi-view RGB dataset, FreiHAND, which
includes hand-object interactions.  However, the annotations are  limited to the
3D poses and shapes of  the hand.  Further, \cite{zimmermann2019freihand} uses a
human-in-the-loop method  to obtain annotations  from multiple RGB cameras  in a
green-screen  background environment.  Our method,  on  the other  hand is  fully
automatic, capable of working even on a single RGBD-camera setup and does not make
any assumption  on the background.  The objects in  our dataset are  also larger
than those  in FreiHAND, thus  resulting in a  more challenging scenario  as the
occlusions are larger. The annotation  accuracy of our method is comparable
to \cite{zimmermann2019freihand} as described in Section~\ref{sec:evalAnno}.

As illustrated  in Fig.~\ref{fig:one} and  Table~\ref{tab:hand_obj_dataset}, our
\datasetname  dataset is  the first  markerless dataset  providing both  3D hand
joints and 3D  object pose annotations for  real images, while the  hand and the
object are heavily occluded by each other.

\begin{table}
  \begin{center}

    \resizebox{.47\textwidth}{!}{%
      \begin{tabular}{lccccccc}
	\toprule
	Dataset & \specialcell{No. of\\Frames} & \specialcell{3D Object\\Pose} & \specialcell{Marker-\\less} & \specialcell{Real\\Images}  & Labels & \specialcell{No. of\\Objects} & \specialcell{No. of\\Subjects}\\
	\midrule
	PAN~\cite{JooSS18} & 675K  & - & + & + & automatic & - & 70\\
	GAN~\cite{Mueller2018} & 300K & - & + & - & synthetic & - & -\\
	FPHA~\cite{garcia2018first} & 100K  & + (23K frames) & - & + & automatic & 26 (4 models) & 6\\
	ObMan~\cite{hasson2019learning} & 150K  & + & + & -  & synthetic & 2.7K & 20\\
	Freihand~\cite{zimmermann2019freihand} & 37K  & - & + & + &  hybrid & 27 & 35\\
	\textbf{HO-3D (ours)} & 78K  & + & + & + &  automatic & 10 & 10\\
	\bottomrule
    \end{tabular}}
    \vspace{-2mm}
    \caption{ Comparison of hand+object datasets.}
    \vspace{-9mm}
    \label{tab:hand_obj_dataset}
  \end{center}
\end{table}

\section{Hand Pose Estimation from Single Color Image}

Fig.~\ref{fig:net_arch} shows the architecture of our hand pose estimator from a
single frame. Given an image of the hand centered in the image window, we first
extract features using the convolutional layers of VGG~\cite{Simonyan14c},  and
then similar to \cite{Cao2017realtime} using a multi-stage CNN, we predict
heatmaps for the 2D hand joint locations and finally joint direction vectors with respect to wrist joint. The hand detection can be done using segmentation as described in Section~\ref{sec:hand_obj_seg}.

\begin{figure*}
	\begin{center}
		\includegraphics[width=0.95\linewidth]{figures/hand_pose_arch.pdf}
	\end{center}
	\vspace{-2mm}
	\caption{Architecture of our hand pose estimator from a single color
          image. Given an input image of hand centered in the image, we extract
          the features using the convolutional layers of VGG~\cite{Simonyan14c}
          (\texttt{Conv1\_1} to \texttt{Conv4\_4}). Similarly
          to~\cite{Cao2017realtime}, we then predict heatmaps for the joint
          locations in multi-stages. The architecture for the different stages are all the same.  \texttt{C} denotes a convolutional layer with the number of filters and the filter size inscribed; \texttt{FC}, a fully-connected layer with the number of neurons; \texttt{P} and \texttt{AP} denote max-pooling and average pooling with their sizes, respectively.}
	\label{fig:net_arch}
\end{figure*}

\section{Hand Pose Estimation for Hand Interaction with Unseen Objects}
Knowing the objects in advance can help to improve the performances of the
estimated 3D hand pose while hand interacts with objects, however, in practice,
the hand can manipulate any arbitrary objects. We have tested our hand pose
estimator trained on our annotations, and tested on sequences where a hand is
manipulating objects not present in the annotated images. As shown in Fig.~\ref{fig:unseen_objects}, our pose estimator performs well on these sequences.

\section{Hand Shape Estimation}
The MANO hand shape parameters $\beta \in \mathbb{R}^{10}$ were estimated for each human manipulator in our \datasetname dataset. The shape parameters are estimated from a sequence $\Phi$ of hand only poses using a method similar to \cite{Tan_2016_CVPR} in two steps. More exactly, the pose of hand $p_h^t$ in the sequence is first estimated for each frame $t$ using a mean pose $\beta_{mean}$ as $\hat{p}_h^{t} = \argmin_{p_h} E_{H}(p_h, \beta_{mean})$, where,
\begin{align}
E_{H}(p_h, \beta_{mean}) = &E_\mathcal{D}(p_h, \beta_{mean}) + \epsilon E_\text{joint}(p_h) + \\ \nonumber
&\eta E_\text{tc}(p_h, p_h^{t-1}, p_h^{t-2}).
\end{align}
$E_\mathcal{D}(p_h, \beta_{mean})$ represents the data term defined in Eq. 2 of the paper where hand is rendered with pose parameters $p_h$ and shape parameters $\beta_{mean}$. $E_\text{joint}$ and $E_\text{tc}$ are explained in Section 3.2 of the paper. At each frame, the pose parameters are initialized with $p_h^{t-1}$.
The personalized hand shape parameters are then obtained as,
\begin{equation}
\beta^* = \argmin_{\beta} \sum_{t\in \Phi} \min_{p_h^t} E_{H}(p_h^t, \beta),
\end{equation}
where the pose parameters are initialized with the values obtain in the first step ($\hat{p}_h^{t}$).

\section{Joint Angle Constraints}
The maximum and minimum limits on the joint angle parameters used in Eq.~(8) of the paper are provided in  Table~\ref{tab:table1}.
\begin{table}
	\begin{center}
		\resizebox{0.47\textwidth}{!}{%
			\begin{tabular}{lccccc}
				\toprule
				Joint & Index & Middle & Pinky  & Ring & Thumb\\
				\midrule
				MCP & \begin{tabular}{@{}c@{}}
					(0.00, 0.45)\\
					(-0.15, 0.20)\\
					(0.10, 1.80)\\
				\end{tabular} & \begin{tabular}{@{}c@{}}
					(0.00, 0.00)\\
					(-0.15, 0.15)\\
					(0.10, 2.00)\\
				\end{tabular} & \begin{tabular}{@{}c@{}}
					(-1.50, -0.20)\\
					(-0.15, 0.60)\\
					(-0.10, 1.60)\\
				\end{tabular} & \begin{tabular}{@{}c@{}}
					(-0.50, -0.40)\\
					(-0.25, 0.10)\\
					(0.10, 1.80)\\
				\end{tabular} & \begin{tabular}{@{}c@{}}
					(0.00, 2.00)\\
					(-0.83, 0.66)\\
					(0.00, 0.50)\\
				\end{tabular} \\
				\hline

				PIP & \begin{tabular}{@{}c@{}}
					(-0.30, 0.20)\\
					(0.00, 0.00)\\
					(0.00, 0.20)\\
				\end{tabular} & \begin{tabular}{@{}c@{}}
					(-0.50, -0.20)\\
					(0.00, 0.00)\\
					(0.00, 2.00)\\
				\end{tabular} & \begin{tabular}{@{}c@{}}
					(0.00, 0.00)\\
					(-0.50, 0.60)\\
					(0.00, 2.00)\\
				\end{tabular} & \begin{tabular}{@{}c@{}}
					(-0.40, -0.20)\\
					(0.00, 0.00)\\
					(0.00, 2.00)\\
				\end{tabular} & \begin{tabular}{@{}c@{}}
					(-0.15, 1.60)\\
					(0.00, 0.00)\\
					(0.00, 0.50)\\
				\end{tabular} \\
				\hline
				DIP & \begin{tabular}{@{}c@{}}
					(0.00, 0.00)\\
					(0.00, 0.00)\\
					(0.00, 1.25)\\
				\end{tabular} & \begin{tabular}{@{}c@{}}
					(0.00, 0.00)\\
					(0.00, 0.00)\\
					(0.00, 1.25)\\
				\end{tabular} & \begin{tabular}{@{}c@{}}
					(0.00, 0.00)\\
					(0.00, 0.00)\\
					(0.00, 1.25)\\
				\end{tabular} & \begin{tabular}{@{}c@{}}
					(0.00, 0.00)\\
					(0.00, 0.00)\\
					(0.00, 1.25)\\
				\end{tabular} & \begin{tabular}{@{}c@{}}
					(0.00, 0.00)\\
					(-0.50, 0.00)\\
					(-1.57, 1.08)\\
				\end{tabular} \\
				\bottomrule
		\end{tabular}}
		\caption{Empirically derived minimum and maximum values for the joint angle parameters used in our implementation.}
		\label{tab:table1}
	\end{center}
\end{table}

\section{Point Cloud from Multiple Cameras}
The $E_\text{3D}$ term in Section~3.2 of the paper uses the combined point cloud $P$ from all the RGB-D cameras. Let $P_c$ denote the point cloud corresponding to camera $c$ and $M_{c_1,c_2}$ denote the relative pose between two cameras $c_1$ and $c_2$. The consolidated point cloud $P$ is then obtained as,
\begin{equation}
P = [P_0,~M_{c_1,c_0}\cdot P_1,~M_{c_2,c_0}\cdot P_2, ... ,~M_{c_{N},c_0}\cdot
  P_{N}] \> ,
\end{equation}
where $[\cdot, \cdot]$ represents concatenation of point clouds.

\section{Hand-Object Segmentation Network}
\label{sec:hand_obj_seg}
The segmentation maps for the hand and object are obtained from a DeepLabV3~\cite{deeplabv2017} network trained on synthetic images of hand and objects. The synthetic images are obtained by
over-laying and under-laying images of hands on images of objects at random locations and scales. We use the object masks provided by~\cite{ycbonline}.
The segmented hands were obtained using an RGB-D camera by applying simple depth
thresholding. We also use additional synthetic hand images from the RHD
dataset~\cite{Zimmermann2017}. A few example images from the training data are
shown in Fig.~\ref{fig:fig12}. We use 100K training images with
augmentations. Fig.~\ref{fig:samples_seg} shows segmentation of hand and object
using the trained DeepLabV3 network.

\begin{figure}
	\begin{center}
		\begin{tabular}{ccc}
			\includegraphics[width=0.32\linewidth]{figures/handObjSeg1} & \hspace{-4mm}
			\includegraphics[width=0.32\linewidth]{figures/handObjSeg3} & \hspace{-4mm}
			\includegraphics[width=0.32\linewidth]{figures/handObjSeg4}
		\end{tabular}
	\end{center}
	\vspace{-2mm}
\caption{Synthetic training images used for training the hand-object segmentation network.}
\label{fig:fig12}
\end{figure}

\begin{figure}
	\begin{center}
	\begin{tabular}{cc}
		\includegraphics[width=0.47\linewidth]{figures/Shreyas_Mustard_1_color_0018} &  \hspace{-4mm}
		\includegraphics[width=0.47\linewidth]{figures/Shreyas_Mustard_1_0018_prediction_graybg} \\
		\end{tabular}
	\end{center}
	\vspace{-2mm}
	\caption{Example  of   hand  and   object  segmentation   obtained  with
		DeepLabV3.  Left: input image; Right: hand (green) and object (purple)
		segmentation.}
	\vspace{-5mm}
	\label{fig:samples_seg}
\end{figure}

\section{Automatic Initialization}
\begin{figure}[ht]
	\begin{center}
		\includegraphics[width=0.8\linewidth]{figures/OS_Ours_compare}
	\end{center}
	\caption{Accuracy of keypoint prediction, described in~Section 3.2 of
          the paper when trained with PAN~\cite{JooSS18} dataset alone and PAN +
          our annotations. The accuracy is measured in percentage of correct 2D
          keypoints given a threshold. Only 15,000 images from our \datasetname dataset are used in training. Due to the presence of object occlusions, a network trained on hands-only dataset is less accurate in predicting keypoints when compared with a network trained with hand+object data.}
	\label{fig:pckCurve}
\end{figure}

As explained in Section~4.1 of the paper, a keypoint prediction network based on convolutional pose machine~\cite{Wei16} is used to obtain initialization for hand poses. Such a network is trained with our initial hand+object dataset of 15,000 images together with images from hand-only PAN~\cite{JooSS18} dataset. Fig. \ref{fig:pckCurve} compares the accuracy of network in predicting keypoints in hand-object interaction scenarios when trained with hands-only dataset and hands+object dataset. Our initial \datasetname dataset helps in obtaining a more accurate network for predicting keypoints and hence results in better initialization.

\section{Dataset Details}
We annotated 77,558 frames of 68 sequences hand-object interaction of 10 persons with different hand shape. On average there are 1200 frames per sequences. 16 sequences are captured and annotated in a single camera, and 52 sequences for the multi-camera setup.

\paragraph{Hand+Object.}
The participants are asked to perform actions with objects.
The grasp poses vary between frames in a sequence in the multi-camera setup and remain almost rigid in the single camera setup.

\paragraph{Participants.}
The participants are between 20 and 40 years old, 7 of them are males and 3 are
females. In total, 10 hand shapes are considered for the annotations.

\paragraph{Objects.}
We aimed to choose 10 different objects from the YCB dataset~\cite{posecnn2018} that are used in daily life. As shown in Fig.~\ref{fig:ycb_objects},
we have a wide variety of sizes such as large objects (e.g. Bleach)  that cause large hand occlusion, or the objects that make grasping and manipulation difficult (e.g. Scissors), while these are not the case in the existing hand+object datasets.

\paragraph{Multi-Camera Setup.} We use 5 calibrated RGB-D cameras, in our multi-camera setup. The cameras are located at different angles and locations. Our cameras are synchronized with a precision of about 5 ms. The scenes are cluttered with objects, and the backgrounds vary between scenes.

Figs.~\ref{fig:HO_3D_1} and \ref{fig:HO_3D_2} show some examples of the 3D annotated frames for both hand and object from our proposed dataset, HO-3D.

\begin{figure*}
	\begin{center}
		\begin{tabular}{cccc}
			\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/00435_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/00556_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/00594_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/00646_concat.jpg} \\

			\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/00746_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/00760_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/00769_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/00787_concat.jpg} \\

			\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/00809_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/00882_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/01002_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/01022_concat.jpg} \\

			\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/01072_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/01108_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/01155_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/01174_concat.jpg} \\

			\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/01353_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/01395_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/01409_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/01791_concat.jpg} \\

			\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/01823_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/01907_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/01994_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/02036_concat.jpg} \\

			\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/02061_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/02107_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/02136_concat.jpg} & \hspace{-3mm}\includegraphics[width=0.24\linewidth,trim=0.8cm 3cm 0.8cm 3cm,clip]{figures/suppDataset/unseenObjects/02195_concat.jpg} \\

		\end{tabular}
	\end{center}
	\vspace{-2mm}
	\caption{Qualitative results of 3D hand pose estimation of hand manipulating unseen objects. Our pose estimator trained on the \datasetname dataset is still able to predict accurate 3D poses when interacting with new objects.}
	\label{fig:unseen_objects}
\end{figure*}

\begin{figure*}
	\begin{center}
	\begin{tabular}{ccc}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/banana/annoVis_01046_3.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/banana/annoVis_01558_1.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/banana/annoVis_01993_0.jpg} \\

		\includegraphics[width=0.33\linewidth]{figures/suppDataset/bleach/annoVis_00495_4.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/bleach/annoVis_01230_0.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/bleach/annoVis_00805_1.jpg} \\

		\includegraphics[width=0.33\linewidth]{figures/suppDataset/cracker/annoVis_0480.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/cracker/annoVis_0809.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/cracker/annoVis_0897.jpg} \\

		\includegraphics[width=0.33\linewidth]{figures/suppDataset/mug/annoVis_01131_1.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/mug/annoVis_01543_2.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/mug/annoVis_1545.jpg} \\

		\includegraphics[width=0.33\linewidth]{figures/suppDataset/mustard/annoVis_0624.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/mustard/annoVis_0833.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/mustard/annoVis_0870.jpg} \\

	\end{tabular}
	\end{center}
	\vspace{-2mm}
	\caption{Some examples of the 3D annotated frames for both hand and object from our proposed dataset, HO-3D.}
	\label{fig:HO_3D_1}
\end{figure*}

\begin{figure*}
	\begin{center}
	\begin{tabular}{ccc}

		\includegraphics[width=0.33\linewidth]{figures/suppDataset/pitcher/annoVis_0393.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/pitcher/annoVis_1004.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/pitcher/annoVis_1736.jpg} \\

		\includegraphics[width=0.33\linewidth]{figures/suppDataset/pottedmeat/annoVis_00431_3.jpg} & \hspace{-3mm}
../		\includegraphics[width=0.33\linewidth]{figures/suppDataset/pottedmeat/annoVis_01133_4.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/pottedmeat/annoVis_01255_1.jpg} \\

		\includegraphics[width=0.33\linewidth]{figures/suppDataset/powerdrill/annoVis_0009.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/powerdrill/annoVis_00563_2.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/powerdrill/annoVis_01039_3.jpg} \\

		\includegraphics[width=0.33\linewidth]{figures/suppDataset/scissor/annoVis_00624_0.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/scissor/annoVis_01617_3.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/scissor/annoVis_01855_2.jpg} \\

		\includegraphics[width=0.33\linewidth]{figures/suppDataset/sugar/annoVis_0500.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/sugar/annoVis_0687.jpg} & \hspace{-3mm}
		\includegraphics[width=0.33\linewidth]{figures/suppDataset/sugar/annoVis_0841.jpg} \\
	\end{tabular}
	\end{center}
	\vspace{-2mm}
	\caption{Some examples of the 3D annotated frames for both hand and object from our proposed dataset, HO-3D.}
	\label{fig:HO_3D_2}
\end{figure*}

\begin{figure*}[ht]
	\begin{center}
		\begin{tabular}{ccccc}
			\includegraphics[width=0.18\linewidth]{figures/ycb/ycb_banana.png} &
			\includegraphics[width=0.18\linewidth]{figures/ycb/ycb_bleach.png} &
			\includegraphics[width=0.18\linewidth]{figures/ycb/ycb_cracker.png} &
			\includegraphics[width=0.18\linewidth]{figures/ycb/ycb_driller.png} &
			\includegraphics[width=0.18\linewidth]{figures/ycb/ycb_mug.png} \\

			Banana & Bleach & Cracker Box & Driller & Mug \\

			\includegraphics[width=0.18\linewidth]{figures/ycb/ycb_mustard.png} &
			\includegraphics[width=0.18\linewidth]{figures/ycb/ycb_pitcher.png} &
			\includegraphics[width=0.18\linewidth]{figures/ycb/ycb_potted.png} &
			\includegraphics[width=0.18\linewidth]{figures/ycb/ycb_scissors.png} &
			\includegraphics[width=0.18\linewidth]{figures/ycb/ycb_sugar.png} \\

			Mustard Bottle & Pitcher & Potted Meat & Scissors & Sugar Box \\
		\end{tabular}
	\end{center}
	\vspace{-2mm}
	\caption{ 10 objects of the YCB dataset~\cite{posecnn2018} that we use for our dataset HO-3D.}
	\label{fig:ycb_objects}
\end{figure*}