\section{Introduction}

Regular examination and skin monitoring is essential for the early detection of cutaneous malignancies including melanoma. Monitoring these pigmented skin lesions over a long period of time is usually a manual, time-consuming and error-prone task, especially when a patient has hundreds of skin lesions, both the inspection and documentation of these lesions are very laborious and inefficient.

Data-driven approaches using dermoscopic images have been developed in recent decades to assist dermatologists in clinical decisions and to spot extremely suspicious situations~\cite{barata2018survey}.
Recent studies have demonstrated the ability of machine learning to match, if not outperform, clinicians in the diagnosis of individual skin lesion images in controlled reader studies where dermoscopic images contain a single skin lesion. For example, algorithms derived from the ISIC Grand Challenge~\cite{rotemberg2021patient} have shown potential to outperform over 500 clinical experts in such a controlled environment~\cite{tschandl2019diagnostic}. However, these studies did not accurately reflect clinical scenarios where clinicians need to examine all skin lesions of a patient regularly.

Despite the potential for using deep learning in clinical dermatology such as skin cancer risk assessment~\cite{de2019development}, most systems assume that the data were adequately pre-selected with all relevant lesions of a patient that are worthy of examination. Thus, the identification of skin lesions in wide-field images (photographs depicting multiple lesions from large body parts) is needed~\cite{birkenfeld2020computer}.

Contrary to classic investigations, where the attention is focused on a particular lesion and a single image taken using a dermoscope or a high-resolution camera, whole-body imaging systems capture images of the whole body. Such a system allows not only lesion detection, but also the possibility of a longitudinal study of dermatological patients. In this manner, it is possible to detect lesions that evolve over time and to find new lesions that emerge after the previous examination.

Traditionally, to monitor all skin lesions on the skin surface of a patient, 2D whole-body imaging has been employed. The use of 2D whole body imaging has shown the potential to reduce the number of biopsies taken, and increase the accuracy of diagnosis for patients at high risk of melanoma~\cite{truong2016reduction}.
Recent advances in artificial intelligence have demonstrated its capability to support physicians in detecting outliers or suspicious ``ugly duckling lesions'' in wide-field images~\cite{soenksen2021using,mohseni2021can}.
A few commercial systems such as Dermengine~\cite{dermengine} support total-body image analysis using mobile cameras.
Nevertheless, the data collection is time and resource intensive and requires a photographer to take photos of subjects in different poses. Thus, skin lesion comparison remains challenging due to changes in the pose or illumination between different scans. The majority of previous registration techniques are evaluated only on a small part of the body (\textit{e.g.} the back or front torso).
For example, a device used for capturing images of the whole area of human skin was introduced in~\cite{strzelecki2021skin}, but the entire body is not captured simultaneously.

Investigations toward the identification of lesions that differ from most of the other marks on a patient’s skin using wide-field images have gained attention to flag them for further examination. Nevertheless, such correlation methods that identify particular lesions of the patient’s body that appear in images taken from different views and time are not well studied in the literature.

Advanced 3D imaging technology, which can be supported by automated analysis, is changing the way to diagnose skin malignancies. 3D imagery has enabled domain researchers to gain new understandings at a faster pace from the image data. 3D data exploration is useful when particular attributes of interests cannot be easily identified in a 2D image, and to accurately inspect the fine topology and textual features of a lesion.

Digital 3D whole-body images enable a complete and automatic documentation of all skin lesions of a patient, determination of their geometric parameters, and their further analysis by a dermatologist~\cite{rayner2018clinical}.
This technology can be used to provide context to single lesion dermoscopy images~\cite{janda2021describing}, facilitate long-term surveillance, and support the evaluation of raised lesions, as the 3D images can be viewed from multiple angles.
This means that lesions are considered within the context of the surrounding skin, offering a more accurate overall assessment and diagnosis than dermoscopy images alone~\cite{grochulska2021additive}.

Preliminary 3D body scanning for detection of new melanocytic lesions is described in~\cite{bogo2014automated}. Such a solution is based on a multi-camera 3D stereo system to capture body shape and skin texture, and a 3D body model to register scans across time. However, the model is based on expensive scanning devices and limited to higher-resolution RGB imagery. The detection of lesions is also based on handcrafted features.

Although there exist numerous recent articles that demonstrated the benefits of 3D whole body photography~\cite{navarrete2020total,betz2021reproducible,grochulska2021additive}, 3D human representation is obtained with expensive commercial systems such as VECTRA{\tiny{\textregistered}}WB360 (Canfield Scientific Inc)~\cite{canfield}, which simultaneously captures 92 images, using them to reconstruct a 3D avatar.
In contrast, we propose a low-cost mobile health pod equipped with AI based smart tools that can be repurposed from current clinician use case, to help researchers collect and curate data automatically from 3D human models as illustrated in Fig.~\ref{fig:3D_collection}.

\begin{figure}[t!]
\begin{center}
\includegraphics[width=0.95\linewidth]{Images/Fig1.png}
\end{center}
\vspace{-12pt}
   \caption{3D Whole body imaging captures nearly the entire skin surface to map, document and monitor pigmented lesions. Visualisation of the sparse point cloud (left), 3D texture model (middle), and skin lesions detected (right). All lesions detected on 2D images are mapped back to the 3D reconstructed model, improving the ability to track lesions over time.
   }
\label{fig:3D_collection}
\vspace{-3pt}
\end{figure}

\begin{table}[t!]
\centering
\vspace{-9pt}
\caption{Comparison of our imaging system to other related works.}
\resizebox{0.8\textwidth}{!}{%
\begin{tabular}{
l c c c c c c
}
\toprule
& ~\cite{zhao2021detection} & ~\cite{strzelecki2021skin} & ~\cite{navarrete2020total} & ~\cite{betz2021reproducible} & ~\cite{grochulska2021additive} & ours
\\
\midrule
2D whole body photography               &            &            & \checkmark & \checkmark & \checkmark & \checkmark \\
Low-cost acquisition system             &            & \checkmark &            &            &            & \checkmark \\
3D human reconstruction                 &            &            & \checkmark & \checkmark & \checkmark & \checkmark \\
Clinical images used for detection      &            & \checkmark & \checkmark & \checkmark & \checkmark & \checkmark \\
Data-driven skin lesion detector        & \checkmark & \checkmark & \checkmark & \checkmark & \checkmark & \checkmark \\
Lesion mapped back on 3D body texture                & \checkmark &            & \checkmark & \checkmark & \checkmark & \checkmark \\
User Interface data management          &            &            & \checkmark & \checkmark & \checkmark & \checkmark \\
\bottomrule
\end{tabular}}
\label{table:literature-summary}
\end{table}

In this paper, we introduce our 3D whole body imaging system to detect and monitor skin lesions. The main specifications of the system compared to the related works are listed in Table~\ref{table:literature-summary}. Our contributions are:
\begin{enumerate}
\item A cost-effective image acquisition system has been designed and constructed, enabling researchers to capture images of the whole body of a patient.
\item A novel 3D pipeline for 3D reconstruction based on open sources tools is proposed to generate high-quality physical-scale 3D body avatars.
\item A processing pipeline that can be used to estimate depth and 3D projection, as well as provide the 3D localisation of skin lesions in whole body images with an optimal balance of inference time and accuracy, has been developed.
\item A custom intuitive and flexible user interface is created that allows users to interact and collaborate with the collected data, the 3D body avatar and detected skin lesions.
\end{enumerate}

\section{Materials and Methods}

In this manuscript, we introduce the 3DSkin-mapper prototype which enables the detection, monitoring and analysis of skin lesions on the patient's entire body.  An abstract view of the system workflow is illustrated in Fig.~\ref{fig:pipeline}.
First, a device is constructed to capture images of the entire body of a patient simultaneously with commercially available digital cameras mounted on a camera rig. These images are then used in a processing pipeline for the 3D image reconstruction, estimation of the 3D depth and 2D to 3D projection. A trained deep learning model localises the lesions within the 2D domain, which are then mapped back to the 3D image of the human body. Such anatomical correspondence of the 3D localisation of all lesions can be exploited for the longitudinal study of the lesions. All information computed in previous stages is integrated into a user interface (UI) which has the function to seemingly integrate several observations from different views into a single point of interest. A key aspect of developing a 3D data collection and curation tool, such as this, is to create an efficient way to associate where a human expert operator is looking with the data itself.
All hardware components used are commercially available off-the-shelf (COTS) and all algorithms developed are based on open sources to facilitate its reproducibility.
Details of each module of the imaging system is described in the following subsections.

\begin{figure*}[!t]
\centering
\includegraphics[width=1\linewidth]{Images/Fig2.png}
\vspace{-9pt}
   \caption{Overview of the 3DSkin-mapper workflow.
  \textbf{1.} Design and construction of the hardware and software for the data collection (60 cameras).
  \textbf{2.} Collection of diverse human data for the 3D reconstruction, 3D fine-tuning and skin lesion detection.
  \textbf{3.} Reconstruction of the human body from different camera poses and sparse point cloud estimation.
  \textbf{4.} Estimation of the depth, human mask and 2D to 3D projection.
  \textbf{5.} A deep convolutional neural network that examines lesions from given wide-field images.
  \textbf{6.} 3DSkin-mapper user interface for visualisation, curation and data management.
   }
\label{fig:pipeline}
\end{figure*}

\subsection{Image acquisition system: hardware and software}
\label{sec:acquisition}

The acquisition system developed consists of 15 poles approximately 2.1 m tall, arranged in a roughly cylindrical configuration with an approximate diameter of 2.2 m. Four Canon EOS 200D cameras are mounted on each pole at the following distances from the ground: 0.3 m, 0.8 m, 1.3 m, and 1.8 m. Black curtains are placed around the camera rig for privacy, improved lighting as well as blocking out the background of images. The cameras are set up to take portrait images with the following settings: aperture measurement (F-Stop) = 10, exposure time = 1/15; sensitivity to light (ISO) = 800; and focal length = 18 mm.
During examination, the patient stands on a motionless wooden platform at the centre of the system, which is used as a reference dimension. This stand is also used to crop and scale the mesh of a person to the right physical height.
The modular acquisition system is shown in Fig.~\ref{fig:sensors_on_trolley}.

\begin{figure}[!t]
\begin{center}
\includegraphics[width=0.75\linewidth]{Images/Fig3.png}
\end{center}
\vspace{-7pt}
   \caption{Whole body photography structure. Camera cage which consist of 15 poles arranged in a cylindrical configuration, each pole with 4 cameras Canon EOS 200D.
   }
\label{fig:sensors_on_trolley}
\vspace{-4pt}
\end{figure}

The Canon EOS 200D cameras used in the system have a 2.5 mm audio jack port for remote triggering. The jack itself has three terminals, which are ground, shutter release, and autofocus, respectively. Connecting either the shutter release or autofocus terminal to the ground terminal will activate the respective function.
Synchronous image capturing is achieved by sending trigger signals to all cameras at the same time. A control board based on FTDI FT232R microcontroller is used for this purpose. The board has 16 3.5 mm audio jack ports for camera connections, and more cameras can be added through purpose-built repeaters. The autofocus signal and shutter release signal are connected to CBUS bit~2 and CBUS bit~3 respectively of the FT232R microcontroller. The signals are set/reset by a Windows application that communicates with the FT232R chip through a USB connection.
Fig.~\ref{fig:trigger_conn} (Top) shows the block diagram of the triggering circuit.

\begin{figure}[!t]
\begin{center}
\includegraphics[width=0.85\linewidth]{Images/Fig4.png}
\end{center}
\vspace{-7pt}
   \caption{Top: Block diagram of the camera triggering circuit. Bottom: Screenshot of the camera control application running on Windows OS.}
\label{fig:trigger_conn}
\vspace{-4pt}
\end{figure}

The autofocus signal is set first, followed by the shutter release signal. $T_{AF}$, $T_{SR}$, and $D_{SR}$ represent the duration of the autofocus signal, the duration of the shutter release signal, and the delay between asserting the two signals, respectively.
These parameters can be configured in the Windows application. By default, they are specified to the following values: $T_{AF}$=2,500ms, $T_{SR}$=3,500ms, and $D_{SR}$=2,600ms.
Fig.~\ref{fig:trigger_conn} (Bottom) illustrates the camera control interface. The drop-down menu in the top-left corner of the GUI lists the USB control boards connected to the PC. There should be exactly one device in the list, which is selected automatically when the GUI is launched. The control signals are sent to cameras by clicking the ``Shoot'' button.
The autofocus signal and shutter-release signal can be enabled or disabled independently by checking/unchecking the ``Focus'' and ``Capture'' boxes correspondingly. This enables different camera settings during image capture. The input boxes next to ``Focus'' and ``Capture'' are used to configure the duration of the autofocus signal and the shutter release signal respectively, while the ``Delay'' input box can be used to configure the delay between asserting the two signals.

The open-source software digiCamControl~\cite{digicamcontrol} is used to change camera settings and transfer captured images from individual cameras to a cloud storage. By launching and properly configuring the software before capturing images, it will automatically download images from all cameras after new images are captured each time.
All cameras and image files are labelled according to their locations, which allows easy mapping of image files to the corresponding camera locations in the rig.

\subsection{Human dataset compilation}
\label{sec:datasets}

\subsubsection{3D body dataset}
Various datasets of 3D body scans are available for research or commercial applications. 3D body scan images with high-quality skin texture information are used to optimise camera poses, and test the processing pipeline. Texture information combined with the 3D geometry allows for advanced analysis and representations.
For rendering realistic images of human bodies, we use 3D human models from Renderpeople \cite{Renderpeople_bundle_swimwear} and the 3DBodyTex~\cite{saint20183dbodytex,saint2019bodyfitr} dataset.
Such human models, as shown in Fig.~\ref{fig:3d_human_models}, also contain skin lesions used for the evaluation of the proposed lesion detector. Synthetic images generated from these models for experiments are obtained using Blender software~\cite{Blender} and Pyrender library\cite{Pyrender}.

\begin{figure}[t!]
    \centering
    \includegraphics[width=0.6\linewidth]{Images/Fig5.png}
    \vspace{-7pt}
    \caption{Synthetic 3D human models. Images adapted from \cite{Renderpeople_bundle_swimwear,saint20183dbodytex,saint2019bodyfitr}.}
    \label{fig:3d_human_models}
\vspace{-4pt}
\end{figure}

\subsubsection{Skin image analysis dataset}
\label{sec:dataset-skin}

Skin lesions are detected directly on 2D wide-field images from publicly available sources, rendered from 3D models and captured by our imaging system. For development and validation of our processing pipeline, we curated a ground truth dataset that contains diverse skin lesions.
Overall, we extracted a total of 24,525 lesion images from the following sources:

\begin{itemize}
    \item We develop an experimental dataset in collaboration with a commercial lab in Australia, which includes de-identified clinical images of skin lesions and ground truth data of the images.
    %
    \item The 2020 ISIC Grand Challenge~\cite{rotemberg2021patient} contains the largest publicly available collection of quality controlled dermoscopic images of skin lesions. This is the official dataset of the SIIM-ISIC Melanoma Classification Challenge hosted on Kaggle~\cite{international2020siim}.
    %
    \item The Edinburgh Dermofit Library~\cite{ballerini2013color} is a collection of focal high quality skin lesion images collected under standardised conditions with internal colour standards. The lesions span across ten different classes including melanomas, seborrhoeic keratosis and basal cell carcinomas.
    %
    \item MED-NODE~\cite{giotis2015med} contains melanoma and nevus images from the digital image archive of the Department of Dermatology of the University Medical Center Groningen (UMCG) in Netherlands.
    %
    \item SD-datasets (SD-198~\cite{sun2016benchmark} and SD-260~\cite{yang2019self}) contains clinical images collected by digital cameras and mobile phones.
    %
    \item PAD-UFES-20 dataset~\cite{pacheco2020pad} consists of samples of six different types of skin lesions. Each sample consists of a clinical image and clinical features including skin lesion location and Fitzpatrick skin type.
    %
    \item Fitzpatrick 17K dataset~\cite{groh2021evaluating} contains clinical images sourced from two online open-source dermatology atlases: DermaAmin~\cite{dermaamin} and Atlas Dermatologico~\cite{dermatologico}.
    %
    \item MIT dataset~\cite{soenksen2021using} contains wide-field images extracted from open-access dermatology repositories, web scraping outputs, and de-identified clinical images from the hospital Gregorio Mara\~{n}\'on (Madrid, Spain). We adopted the clinical images provided as supplementary material.
    %
    \item DermNet NZ~\cite{dermnet} is supported and contributed by New Zealand Dermatologists on behalf of the New Zealand Dermatological Society Incorporated.
    %
    \item 3DBodyTex~\cite{saint20183dbodytex,saint2019bodyfitr} consists of 400 high resolution 3D texture scans of 100 male and 100 female subjects captured in two different poses. Some pigmented lesions are visible on the skin.
\end{itemize}

\begin{figure}[t!]
\begin{center}
\includegraphics[width=0.8\linewidth]{Images/Fig6.png}
\end{center}
\vspace{-10pt}
   \caption{
   Selected samples of typical whole-body photography included in the skin image analysis dataset. Images are from our dataset and available in~\cite{rotemberg2021patient,soenksen2021using,dermnet}
   }
\label{fig:dataset-collected}
\vspace{-4pt}
\end{figure}

To improve the performance of the detector, we also adopted GAN-based data augmentation for skin lesion analysis~\cite{bissoto2021gan}.
To generate synthetic skin lesion images, we adapted the conditional GAN (cGAN) proposed in~\cite{pix2pix2017} which has been successfully used in many cross-domain learning tasks.
Selected samples of raw images that were used to train the skin lesion detector are illustrated in Figure~\ref{fig:dataset-collected}.

\subsubsection{Whole body image captured from participants}
Using the proposed imaging system described in Subsection~\ref{sec:acquisition}, we collected images from participants to test the 3DSkin-mapper workflow.
With a single trigger, sixty images were captured by the 60 cameras from different angles simultaneously. These images represent different views of the human body, and were used to reconstruct a 3D human avatar for visualisation, and 3D mapping of skin lesions for computer-aided skin cancer diagnosis. Selected views of a participant are illustrated in Fig.~\ref{fig:participant_data}.

\begin{figure}[t!]
\begin{center}
\includegraphics[width=1\linewidth]{Images/Fig7-min.png}
\end{center}
\vspace{-14pt}
   \caption{Sample of collected data from a participant with the proposed whole-body imaging system. Selected poles from the imaging system are shown as orange circles.
   }
\label{fig:participant_data}
\vspace{-8pt}
\end{figure}

\subsection{3D human reconstruction}
\label{sec:3Dreconstruction}

\subsubsection{Whole body image rendering from 3D human models}
The viewing angles, positions and field of view (FoV), determined by camera focal length) of captured images could significantly affect the quality of the 3D reconstructed human avatar. Therefore, the angles and positions of the cameras need to be adjusted as much as the setup discussed in Subsection~\ref{sec:acquisition} allows until the best possible quality of images is achieved. To save time, instead of testing different camera positions on the camera rig, we simulated this process by generated synthetic images using computer graphic rendering with similar configuration as our camera rig. For every camera configuration, a set of 60 synthetic images are rendered from an existing 3D human model. The images are processed to reconstruct a 3D human model which is visually inspected for quality.

There are two conflicting parameters to compromise: i) a wider camera field of view provides a more complete 3D model and more image overlapping between nearby views provides better estimations of camera poses; ii) image resolution for accurate depth estimation for 3D reconstruction.
A camera pose represents the 3D position and orientation of the camera that captures an image. Incorrect camera poses as well as less view overlapping lead to noisy depth and a low-quality 3D reconstructed model.
In practice, we mainly try to make sure all cameras are detected and optimised, and minimise the missing part of the output 3D model, particularly the head, shoulder, hands and feet.

Blender~\cite{Blender}, a popular 3D computer graphics open-source software with a Python API, was firstly used to render a 3D human model in a controlled environment. However, the rendering time was too long and repeated texture of the environment could cause errors in camera pose estimation during 3D reconstruction.
Thus, we adopt Pyrender~\cite{Pyrender}, a light-weight Python package, to render the 3D model only without background scene. This can speed up the rendering while producing good quality rendered images for testing the proposed processing pipeline.

\subsubsection{Image acquisition system evaluation}
To reconstruct a complete and accurate 3D human model, we use optimal positions and angles of the cameras from the virtual cameras used to render synthetic images.
Besides the attention to the camera positions, angles and FoV, camera settings including ISO (that is, the gain of the camera sensor), exposure time, aperture and focus also affect the quality of the images.
Higher gain leads to a brighter but also noisier image. Exposure time represent the amount of light captured by the camera sensor. Longer exposure time leads to a brighter image, but subject to motion blurs. Too high ISO or too long exposure time also lead to image saturation, where image details are lost in too bright image regions. As a result, a balance between ISO and exposure time is determined to obtain the best image quality for a given scene and lighting conditions. As depth of focus is finite and therefore only part of the object is in focus, the focus of the camera lens needs to be adjusted to maximise the image sharpness of the body part to be reconstructed. Bright, no saturation, low noise, and sharp images will lead to more successful 3D reconstruction.

By optimising these camera internal settings, we gradually improve the quality of the 3D reconstructed models from real images. Multi-view stereo reconstruction (MVS) successfully detected the camera poses of all 60 images to create a 3D human model as shown in Figure \ref{fig:meshroom_vs_capturereality} by two different software solutions. Although the head of the person is incomplete in regions where no lesions are detectable (hair / scalp), the 3D reconstructed mesh is good enough for skin analysis.

\begin{figure}[t!]
    \centering
    \includegraphics[width=0.75\linewidth]{Images/Fig8.png}
    \caption{Meshes reconstructed by Meshroom (left) and Reality Capture (right).}
    \label{fig:meshroom_vs_capturereality}
    \vspace{-6pt}
\end{figure}

\subsubsection{3D reconstruction approach}
3D avatars of participants are generated from 3D total body photography for lesion identification with deep neural networks.
For this purpose, two 3D reconstruction software packages were evaluated:
i) Meshroom \cite{Meshroom, griwodz2021alicevision} is a powerful open-source software package that provides very flexible environment for customisation running on Linux and Windows environments;
ii) Reality Capture \cite{RealityCapture} is a well-know commercial software which provides simple and high-resolution 3D reconstruction running on Windows environment only.
Fig.~\ref{fig:meshroom_vs_capturereality} illustrates the 3D meshes reconstructed using Meshroom and Reality Capture, respectively.

Both software packages produce decent quality meshes, however missing a portion of the top of the head. Reality Capture clearly produces a mesh with higher resolution with slightly more completeness. However, Reality Capture was not able to detect camera poses of four images and excluded them, while Meshroom could detect camera poses for all 60 images. Due to the flexibility and open-source nature of Meshroom, we adopted this tool in the 3DSkin-mapper workflow. For this purpose, we customised a 3D reconstruction pipeline for the implementation of ground plane detection, automatic cropping, scaling and alignment as follows:

\begin{figure}[t!]
    \centering
    \includegraphics[width=1\linewidth]{Images/Fig9.png}
    \caption{Meshroom's standard 3D reconstruction pipeline with solid arrows. The custom reconstruction pipeline is shown by the dotted arrows to align, crop and scale the 3D mesh.}
    \label{fig:pipelines}
    \vspace{-6pt}
\end{figure}

\begin{enumerate}
    \item \textit{Standard reconstruction pipeline}. Such a pipeline is illustrated in Fig.~\ref{fig:pipelines} following the solid arrows. The main steps are structure-from-motion, depth map estimation, meshing and texturing.

    \item \textit{Custom reconstruction pipeline}. The custom 3D reconstruction adds new functionalities including finding the ground (by plane fitting to the reconstructed point cloud), the stand and the scale (by circle fitting) and the transformation of the reconstructed body (by setting the centre and normal vector of the stand to be the world centre and vertical axis respectively).
    %
    Fig.~\ref{fig:overlay} shows the output of structure-from-motion (left), and the filtered non-textured mesh (right) produced by the standard reconstruction. Notice that the non-textured mesh contains the body standing on the floor and the curtain around, and these are removed in the textured mesh of the person.
    %
    The 3D origin is on the ground between the feet, the ground normal represents the y-axis, and the height of the body matches with the physical height of the person. The stand provides the scaling factor from the known diameter, and the origin from the fitted circle center. Such a proposed pipeline is shown as dotted arrows in Fig.~\ref{fig:pipelines}.
    %
    Qualitative results of the 3D whole body skin surface scans and the 3D model with the customised pipeline is depicted in Fig.~\ref{fig:3D_collection}.
\end{enumerate}

\begin{figure}[t!]
    \centering
    \includegraphics[width=1\linewidth]{Images/Fig10.png}
    \caption{Camera poses and a sparse point cloud estimated (left), and reconstructed and filtered mesh (right) of the whole scene. The finale aligned, cropped, scale and textured mesh of the person is shown as a small upright coloured 3D model near the middle. The final 3D model is overlaid to together with the filtered mesh to show the differences in size, scale and orientation.}
    \label{fig:overlay}
    \vspace{-6pt}
\end{figure}

\subsection{3D Data pre-processing}
\label{sec:preprocessor}

Given the reconstructed 3D model and camera intrinsic and extrinsic parameters, a \textit{preprocessor} renders the 3D models to get the corresponding depth image to each captured image, and a mask indicating which pixels are associated with the subject for each camera view (see Fig.~\ref{fig:preprocessor}). 2D to 3D projection can then be obtained from the depth image and camera parameters.
The process of 2D to 3D mapping of skin lesions is performed as follows:

\begin{enumerate}
    \item The model view projection (MVP) matrix is calculated from the estimated camera intrinsics / extrinsics
    \item A scene is constructed consisting of the reconstructed 3D mesh and a single ambient light source.
    \item The scene is rendered using the Pyrender (OpenGL) library to produce a colour and depth image from the camera's viewpoint.
    \item For each pixel in the image, a 3D ray is calculated from the MVP matrix. The ray is combined with its associated depth value to produce the 3D coordinates associated with each pixel.
    \item Every pixel which has an associated 3D coordinate within the predefined capture region (cylinder with origin in the centre) is marked as containing the subject.
\end{enumerate}

\begin{figure}[t!]
    \centering
    \includegraphics[width=1\linewidth]{Images/Fig11.png}
    \caption{The input view of preprocessor (left), rendered 3D mesh (middle left), depth image (middle right) and subject mask (right).}
    \label{fig:preprocessor}
    \vspace{-4pt}
\end{figure}

A small mismatch is present between the input camera view and the rendered outputs becoming more pronounced near the edges of the images. This is caused by the estimated camera model not accounting for distortion in the camera lens. Such condition has no effect in this application because the skin lesions are detected using the rendered 2D images.

\subsection{Skin lesion detection}
\label{sec:lesion-detector}

Human skin lesions are not homogeneous, which include structures such as pathological and benign melanocytic lesions, wounds and scars. To identify such lesions, we propose a method based on deep learning object detection as such methods have been useful for practical real-world skin lesion observations~\cite{soenksen2021using}. These models indicate the location and dimension (height and width) within the 2D domain (2D images collected with the whole body imaging system). Then, these detected skin lesions can be mapped back to the 3D surface of the subject, with the associated 3D coordinate of every pixel estimated with the preprocessor described in Subsection~\ref{sec:preprocessor}.

Object detectors have achieved exceptional performance in recent years, thanks to advances in deep convolutional neural networks. The model designs are broadly organised in two categories: one-stage and two-stage detectors.
One-stage detection frameworks, use a single network stage to perform classification and bounding box regression. Two-stage detection frameworks, on the other hand, includes a pre-processing step for generating object proposals to identify the object class and to regress an improved bounding box.

\begin{figure*}[!t]
\begin{center}
\includegraphics[width=1\linewidth]{Images/Fig12.png}
\end{center}
\vspace{-12pt}
   \caption{Structure of object detectors, highlighting the main components of the Backbone, Neck and Head, and how information flows between these components. Image recreated from~\cite{bochkovskiy2020yolov4}.
   }
\vspace{-4pt}
\label{fig:overview_det}
\end{figure*}

Variants of models are based on changes on the main components of the standard architecture of an object detector. The structure for two-stage and one-stage object detectors are illustrated in Fig.~\ref{fig:overview_det}.
Common components that may be changed are: a \textit{backbone} used as feature extractor, a \textit{neck} used to extract different feature maps from different stages of the backbone, and a \textit{head} which is responsible for the detection of bounding boxes. Such a head can be a dense prediction (one-stage) or a sparse prediction (two-stage)~\cite{liu2020deep,sharma2020comprehensive}.

We opt to evaluate the performance of one-stage detectors as they are better suited and highly viable for real-world practical applications due to the faster inference.
Among well-known one-stage object detectors, the scaling cross stage partial network (\textit{Scaled-YOLOv4})~\cite{wang2021scaled} from the YOLO family of detectors (You Only Look Once~\cite{redmon2016you}), can achieve an optimal trade-off between speed and accuracy. Another important reason that this model was adopted is that it can be used for commercial purposes.

Scaled-YOLOv4 makes an incremental improvement to YOLOv4~\cite{bochkovskiy2020yolov4} with a network scaling approach that modifies not only the network depth, width, and resolution; but also the structure of the network.
The model is composed of an optimised CSPDarknet53 as a backbone, which employs a Darknet-53~\cite{redmon2018yolov3} and a CSPNet strategy~\cite{wang2019cspnet} to partition the feature map of the base layer into two parts and then merges them through a cross-stage hierarchy. A spatial pyramid pooling (SPP)~\cite{he2015spatial} additional module and a path aggregation network (PANet)~\cite{liu2018path} with CSP are used as neck. Finally, a YOLOv3~\cite{redmon2018yolov3} is used as the head.

With the refined whole body imaging system, a set of images and their corresponding camera poses are estimated to test the skin lesion detector and 2D-3D mapping.
The 3D reconstruction from synthetic images provides the estimated camera poses, depth and mask of each input image. These are used to compute the 3D position of each detected lesion from a 2D image onto the 3D model for management and longitudinal tracking.
We map each centre point of the predicted 2D bounding box coordinates to the 3D space. As one lesion can appear multiple times across multiple images, skin lesions that have similar 3D positions on the 3D model are grouped as a specific global lesion ID. For this purpose, a thresholded distance and an agglomerative clustering technique of all 3D lesions are computed.
Fig.~\ref{fig:global_ID} illustrates the different views of two selected skin lesions and their representations in the 3D space.

\begin{figure*}[!t]
\begin{center}
\includegraphics[width=0.85\linewidth]{Images/Fig13.png}
\end{center}
\vspace{-12pt}
   \caption{Visualisation of two selected skin lesions and their different views from each camera. Lesions visible from poles C, B, A, O, N, and M.
   }
\label{fig:global_ID}
\vspace{-6pt}
\end{figure*}

\subsection{3DSkin-mapper UI}
\label{sec:ui}

We developed a custom user interface (UI) to visualise our results. The system centres around multiple linked views of the captured and processed data. Such interface is illustrated in Fig.~\ref{fig:qualitative_results_interface}.

A thumbnail grid view of the images that were captured in the imaging system (See Fig.~\ref{fig:participant_data}) appears at the bottom of the central column of the interface. Users can filter the visible thumbnails based on the area and pole position the images were captured from (\textit{e.g.} Front view: Poles C, B, A, O and N).
The selected image appears in the central top view of the interface and allows the users to zoom and pan around the image for more detailed inspection. The rectangular region of the detected features can be shown as an overlay on the central primary image. An individual detection can be selected within the bounding box region of the detection on the central primary image.

\begin{figure}[!t]
\begin{center}
\includegraphics[width=0.96\linewidth]{Images/Fig14.png}
\end{center}
\vspace{-12pt}
   \caption{Description of the main components in the interface for data management.
   }
\label{fig:qualitative_results_interface}
\vspace{-6pt}
\end{figure}

The left side of the interface shows the reconstructed textured 3D model (See Subsection~\ref{sec:3Dreconstruction}). The view of the 3D model can be altered with the mouse, allowing the model to be rotated around roughly the centre of the model (mid-abdomen), as well as manipulating the camera around the model (pan / tilt as well as moving the camera in/out). There are also user interface options to align the 3D camera view to a fixed predefined position pointing towards the model, as well as moving the model back to the centre of the viewport (while maintaining the current camera position). The model appears with textures by default, but the textures can be turned off to allow more detailed inspection of the 3D mesh without the added visual cues from the textures, which mask some underlying 3D model imperfections.

The right panel highlights the active detection (top-right) showing either the raw image pixels of the detection enlarged with smoothing optionally applied to facilitate clinical use cases. Alternate feature detection crops from the current image appear in the ``Current Image Detections'' image grid. Finally, the ``Current Detection Metadata'' region allows the current selected feature detection to be annotated with metadata (\textit{e.g.} generic textural data that can be expanded to more rich metadata fields).
Further, we allow for curation of the dataset by annotating misclassifications with the ``Remove Detection'' button (which are then removed from visualisation on the user interface). Selecting the active detection (either by selecting within the bounding box region of detection directly from the central primary image or the ``Current Image Detections box'') updates the selection across all views (highlighting on the 3D model, primary image, enlarged image crop and selected current image detection and metadata).
When the actively selected detection is changed, the virtual camera viewing the 3D model can optionally be moved to a fixed offset from the 3D point on the mesh surface of the model to point directly at the computed 3D points of the detection (focusing the 3D view on the actively selected detection). This is achieved by offsetting the camera a fixed distance from the computed 3D coordinate of the detection on the model surface along the computed normal vector at this location.

The UI was developed so that it was agnostic to the underlying model generation, skin lesion detection and geometric transformation computations. This completely decouples the data processing from the user interface presentation, by using directories of flat files as the transport mechanism between these separate systems. This means that the UI is a modular product that could be reused for a different application by simply feeding in the appropriate model, images, and detection metadata.

The UI was developed using QT via PySide2 bindings from Python version 5.15.2. The UI takes as input the 60 images saved from the DSLR cameras, the 3D model (stored in an .obj file with an associated single texture image) and a JSON data file which contains the information on the detections (\textit{e.g.} unique identifier, 3D coordinate location and 3D normal direction). Linkage of the data in the JSON file to the underlying image in which the detection was found can be made via filename correspondence (prefixed the same - different file extension) or a metadata field which stores the original image which the detections in the file came from.

\section{Results and discussion}

\subsection{3D model reconstruction}

Validation of 3D reconstruction accuracy is performed by computing the Hausdorff distance (using MeshLab~\cite{cignoni2008meshlab}) of 3D reconstructed meshes relative to the groundtruth 3D meshes. Hausdorff distance is defined as the largest distance between two meshes. So when comparing two meshes, large Hausdorff distances indicate missing features in the reconstruction results. Once the Hausdorff distance was computed and normalised by the diagonal distance of the 3D mesh bounding box, it is visualised as a heatmap on the ground truth meshes.

The results and their comparison to the ground truth 3D model are shown in Fig.~\ref{fig:comparison_groundtruth_reconstruction}. Significant errors occur in the regions of underarm and upper shoulder due to limited image coverage there.

\subsection{Lesion detection}

\subsubsection{Evaluation protocol and experimental setup}

The reliability of lesion detection is evaluated by comparing the obtained detection results with ground truth images manually annotated with 2D bounding boxes. A scale-invariant feature transformation (SIFT) is used first to detect all blob-like regions (group of connected pixels in an image that share some common property) to accelerate the annotation process. Lesions that occupy an area of less than $25 \times 25$ pixels (approximately $\leq$ 2 mm diameter) are not included, as these lesions cannot be reliably detected with the current quality of the wide-field images. To estimate the diameter, Otsu thresholding is used.
Difficulty in obtaining high enough resolution images for automated detection of small lesions, has also led to issues in obtaining accurately labelled lesions in the training images.

We use the normalised intersection-over-union (IoU) metric to define how well a bounding box overlaps the associated ground truth. We adopt the mAP@IoU[0.5], where the mean average precision requires that the bounding boxes overlap with an IoU $\ge$ 0.5. We also report recall and precision. Given a predefined confidence threshold, precision defines the number of true positives produced relative to the total of the true positive plus false positive, and recall defines how many of the ground truth instances are correctly classified relative to the total number of ground truth instances.
We calculated each metric for all whole body images and report the average over all images.

\begin{figure}[!t]
\begin{center}
\includegraphics[width=1\linewidth]{Images/Fig15.png}
\end{center}
\vspace{-9pt}
   \caption{Comparison between the ground-truth human model (left) with the reconstructed model (middle). The reconstruction error in terms of Hausdorff distance is shown as colour or map and histogram (right). We can see some missing parts on the hands and underarms, and the average error is around 2 mm.
   }
\label{fig:comparison_groundtruth_reconstruction}
\vspace{-6pt}
\end{figure}

For the training and testing of the skin lesion detector, the whole body images are split into patches of $608 \times 608$ pixels with a 50\% overlap. The detections are then aggregated across all patches and non-max suppression (soft-NMS~\cite{bodla2017soft}) is performed to obtain the final set of lesion detections for the entire input image.

The skin lesion detector is trained using the collected dataset described in Subsection~\ref{sec:dataset-skin} (whole body images spanning various skin-types and body locations) with random split into training (60\%), validation (20\%), and testing (20\%) sets. The model is trained with Mosaic data augmentation, batch size 64, learning rate 1$e^{-3}$, and implemented with the Darknet deep learning framework available in~\cite{darknet-ab}.

\begin{figure}[!t]
\begin{center}
\includegraphics[width=0.9\linewidth]{Images/Fig16.png}
\end{center}
\vspace{-9pt}
   \caption{Qualitative results of the skin lesion detector on clinical images introduced in Subsection~\ref{sec:dataset-skin}.
   }
\label{fig:qualitative_results_skin detector}
\vspace{-6pt}
\end{figure}

\subsubsection{Detecting lesions on 2D images and 3D mapping}

The trained model reaches an mAP@IoU[0.5] of over 92\% with a precision of 0.79 and a recall of 0.88 which is an accurate overlap of bounding boxes by current standards in the literature. Fig.~\ref{fig:qualitative_results_skin detector} illustrates some qualitative results of the skin lesion detection. Our algorithm automatically detects skin lesions larger than around 2.5 mm diameter and provides an opportunity to perform quick screening for skin lesions in whole body images. This is especially useful for patients with many lesions.

As described in Section~\ref{sec:lesion-detector}, the system categorised the same skin lesion detected in multiple 2D images into a unique 3D global lesion ID. This functionality allows the user to visualise the lesion from different perspectives, which would allow operators to overcome a potential shortcoming from any single image (such as a partial occlusion). Fig.~\ref{fig:qualitative_results_3D map} shows the lesions detected on 2D images, as well as the mapping of these lesions onto the 3D model reconstructed.
Limitations of our model include the difficulties to filter the lesion according to more advanced and robust geometrical parameters. By incorporating an additional data-driven segmentation module (lesion / background segmentation) for each lesion contained in a bounding box, the system will be able to rank all the lesions with such conditions. The correct diameter estimation is especially important when comparing the images captured over a period of time. A significant increase in the diameter of a nevus between consecutive examinations is a strong indicator that the melanocytic nevi can be potentially dangerous.

\begin{figure}[!t]
\begin{center}
\includegraphics[width=0.8\linewidth]{Images/Fig17-min.png}
\end{center}
\vspace{-9pt}
   \caption{Visualisation of the selected skin lesions detected in the 2D image (left) and mapped back to the 3D surface of the subject (right). To facilitate visualisation of the 3D model, half of the human body is represented without texture.
   }
\label{fig:qualitative_results_3D map}
\vspace{-6pt}
\end{figure}

\subsubsection{Tracking of skin lesion across temporal scans}
For the purpose of longitudinal tracking of skin lesions (\textit{i.e.} to detect new lesions and temporal changes of prior lesions), a patient can be scanned at different times to form a set of 3D meshes. Thus, variations in the pose across scans are required for the matching of skin lesions.
For this purpose, we first determine anatomical correspondence across 3D human meshes, then, we establish correspondence between the lesions of any two meshes~\cite{bogo2014automated,korotkov2018improved} (\textit{e.g.} we map the centre point of the predicted 2D bounding box coordinates to the corresponding temporal adjacent 3D mesh).
The 3D information is used for correspondence matching as the visual appearance of the lesions change over time in different scans, which is a limitation of early methods based on only 2D anatomical landmarks and structured graphical models~\cite{mirzaalian2016skin}.

\begin{figure}[!t]
\begin{center}
\includegraphics[width=0.9\linewidth]{Images/Fig18.png}
\end{center}
\vspace{-13pt}
   \caption{Skin lesion matching across scans with different poses.
   Top: Skin lesion detection in the 2D image (t=15 days).
   Bottom: 3D meshes and longitudinal tracking of selected skin lesions across different sessions (t=0; t=15 days; t=30 days). The coloured line between each lesion indicates the unique lesion correspondence across different images.
   }
\label{fig:skin_tracking}
\vspace{-6pt}
\end{figure}

Learning-based deformation methods can be exploited for 3D correspondence between two 3D meshes~\cite{deprelle2019learning}. This means that captured skin textures are in accurate alignment across scans, facilitating the detection of changing lesions~\cite{zhao2021detection}.
We adopted the trained network LoopReg~\cite{bhatnagar2020loopreg} on the Faust dataset~\cite{bogo2014faust} (which contains around 100 scans of undressed people) for correspondence prediction of vertices across scans of the same subject. Such a model does not require instance specific annotations, supports self-supervision and does not require multiple initialisation in comparison with baseline models such as 3D-CODED~\cite{groueix20183d}.
Finally, with the vertices of the registered meshes, we solve the matching problem based on 3D geodesic distance between pairs of lesions, where the corresponding lesion will be located at the more proximate anatomical location of the target mesh.
In this experimental phase, newly-appearing and disappearing skin lesions were not possible to evaluate.
Fig.~\ref{fig:skin_tracking} shows qualitative results of the longitudinal testing.

3D whole body imaging is notably important in the analytical assessment of individual characteristics by evaluating individual lesions relative to their overall skin lesion ecosystem. Such technology is useful to track lesions of patients with many skin lesions, where diagnostic biopsy of every lesion is unfeasible.
Our low-cost imaging system provides a promising tool to capture whole body images, detect and document lesions. This can potentially save lots of time of skin cancer specialists. 3D imaging can facilitate long-term surveillance by monitoring the entire skin surface rather than individual lesions.

\section{Conclusion}
\label{conclusion}

In this study, we introduced a 3D whole body imaging system and workflow for multiple view image capturing, 3D human body reconstruction, 3D skin lesion localisation and documentation. A software tool has been designed for data curation and management. We have also developed an algorithm to quickly identify skin lesions in whole body images. Software functions are developed based on open sources such as digiCamControl, Meshroom, Darknet, and QT via PySide2 for camera control, 3D human body reconstruction, lesion detection, and user interface development, respectively.
Future work will involve further evaluation of the workflow against a larger image set of expertly labelled images.

\bibliographystyle{splncs04}


\section{INTRODUCTION}

	Suction cups, widely used in logistics, are robust and simple
	but usually only suited for grasping objects from the top and restricted by the presence of a surface appropriate for a vacuum seal.
	Dexterous hands are capable of complex manipulations, but are also expensive and difficult to control,
	requiring careful consideration of their purpose before their adoption \cite{grebensteinAntagonisticallyDrivenFinger2010, akkayaSolvingRubikCube2019}.

	For this reason, simple structures such as two-finger grippers tend to be widely used. However, unlike suction cups, grippers need to be properly aligned to the object to be grasped.
	%

	Under teleoperation, this positioning can be stressful and tiring, particularly when done over long periods of time.
	For example, when grasping a small object such as a screw on a desk with an adaptive finger, the operator needs to precisely predict the position of the fingertip before the finger is closed to avoid hitting the desk while assuring a successful grasp. This procedure requires careful and difficult adjustment of the spatial position of the finger often without visual references.
	In fact, it has been shown that teleoperation can take time to perform even on simple movements such as the pick up of a straw \cite{li_teleop_2017}.

	\begin{figure}
		\centering
		\vspace{0.2cm}
		\includegraphics[width=0.9\linewidth]{intro_light}
		\vspace{-0.2cm}
		\caption{
				The F1 hand and some of its fingers.
				(a) The absence of actuation allows for easy shaping and switching of the fixed finger.
				(b) The F1 hand attached to a Toyota's HSR is capable of grasping objects as thin as a 0.7 mm thick clip.
				(c) We propose a control strategy to recover actuation symmetry as the atypical structure changes the origin of the gripper's tip reference frame  ($O_{tip}$) from the object center.
				Experiments can be watched by following the link: \href{https://youtu.be/iWXXIX4Mkl8}{https://youtu.be/iWXXIX4Mkl8}.
		}
		\label{fig:intro}
	\end{figure}
	To overcome such problems, we investigate a two-finger gripper with only one actuated finger, named F1 hand.
	By using the fixed finger as a reference, the operator can easily adjust the position of the fixed finger alongside the target object, which enables a more intuitive grasping in teleoperation.
	%

	{\color{black}
		Moreover, the adoption of a fixed finger not only decreases the cost of the hand, but it can be easily switched as the finger is free from actuators and electronic circuits. By tailoring the fixed finger it is possible to greatly expand the grasping capabilities of the F1 hand as shown in Fig. \ref{fig:intro}(a).
	}

	However, as the closing motion of the fingers are asymmetrical, the control method for autonomous grasping with the F1 hand is not obvious, precluding one from using existing grasping methods for symmetrically actuated grasps.
	As such, we propose a grasping strategy to enable the use of  existing grasping methods for symmetric grippers
	by exploiting the entire arm motion during grasping {\color{black}(Fig. \ref{fig:intro}(c))}.

	For the evaluation of the resulting system, many studies evaluate the usefulness of the hand separately from the robot body \cite{grebensteinAntagonisticallyDrivenFinger2010, maM2GripperExtending2016,  achilliDesignSoftGrippers2020, KoTendonDrivenPFN} or tested with a prescribed motion of the robot arm in a predefined environment \cite{ bircherComplexManipulationSimple2021, yaguchiDevelopmentAutonomousTomato2016}.
	However, in practice, it is not always possible to set the robot hand and body at a predefined position in a real environment.
	To prove the practical usefulness of the F1 hand and its control methodology for autonomous grasping, we investigate the total grasping performance including the interaction between the robot’s motion and the gripper’s function.

	\begin{comment}

	Our contribution is twofold.
	We propose the F1 hand where one of the two fingers is fixed.
	It enables much intuitive grasping in teleopration by letting the operator directly control the position of the fixed finger alongside the target object. We do not need to predict the finger movement.
	It is also structurally simple and low-cost, and has high versatility because the fixed fingers can be manufactured in various shapes for easy switching (Fig. \ref{fig:intro}(a)). Because the movable fingers are constructed with an adaptive function, the grasping form can be automatically changed according to the shape of the object, so it is possible to grasp various objects from a small paper clip to a bulky hand drill.
	A second contribution addresses the control challenges introduced by the atypical asymmetric design of the F1 hand.
	To this end, we propose a \emph{grasping controller} that uses the whole arm/robot movements to compensate for the asymmetric closing of the gripper (Fig. \ref{fig:intro}(b)).
	We show that the grasping controller allows us to use the hand as if it was symmetrically actuated by directly feeding the results of an off-the-shelf method for grasp pose estimation.
	\end{comment}

	\section{RELATED WORK}
	There is a vast amount of research on robotic hands, with a wide variety of types and features.
	Even if we constrain the literature to only two-finger designs, the body of work can range from soft grippers with multi-segments \cite{teeple2020_hand} to designs specific for thin \cite{yoshimi2012_hand} and  small \cite{watanabe2021_hand} objects. In this section, we will analyze the characteristics of a typical two-fingered gripper and a gripper with one finger fixed.

	\subsection{Two-Finger Grippers}

	\begin{figure}
		\centering
		\vspace{.2cm}
		\includegraphics[width=0.9\linewidth, draft=false]{gripper_type}
		\caption{
			Three different types of two-finger gripper designs. From left to right, a parallel gripper, a multi-linkage adaptive finger, and our proposed F1 hand with a fixed finger.
			The arrows illustrate the motion of the tip of the finger.
		}
		\label{fig:gripper_type}
	\end{figure}

	As illustrated in Figs. \ref{fig:gripper_type}~(a,b), for our study, we categorize such grippers by the motion of the finger as grippers with parallel sliding fingers and multiple-linkage adaptive fingers.
	Parallel grippers are characterized by having the closure of the fingers done via linear motion and the grasping force is usually due to compression forces.
	Although some designs exist that eliminate this limitation (e.g.~\cite{kobayashiDesignDevelopmentCompactly2019}), parallel grippers usually have a limited maximum stroke as the amount of finger travel directly affects the size of the hand.
	For example, the Robotiq Hand-E has a stroke of 50 mm while the standard gripper of the Franka Emika has a stroke of 80 mm.

	Multiple linkage fingers offer a better ratio between the size of the hand and the finger aperture.
	The multiple segments of the fingers are effective for grasping via robust mechanical enveloping (Fig. \ref{fig:gripper_type}(b)).
	Examples of such grippers are the Robotiq 2F-140, the Kinova gripper KG-2, the Toyota Human Support Robot (HSR) platform \cite{yamamoto2019development}, and our proposed gripper, presenting an opening distance between fingers of 140 mm,  175 mm, 125 mm, and 130.5 mm, respectively.
	Thus, a hand with a linkage mechanism can achieve a larger opening than a parallel gripper, but on the other hand, multiple linkage fingers make it difficult to intuitively know where the fingertip will be located when the finger is closed (Fig. \ref{fig:hsr_f1_z_height}).

	\begin{comment}
	An issue of multiple-linkage fingers is that the distance between the wrist and the fingertips increases as the fingers are closed due to the rotation of the links (the thick arrows in Fig. \ref{fig:gripper_type}~(b)).
	This variable distance  leads to the problem shown by the upper sequence of snapshots in Fig. \ref{fig:hsr_f1_z_height} where the teleoperator has to simultaneously adjust the closing of the fingers while lifting the arm to avoid the fingers pressing too hard on the surface.
	In autonomous grasping, the bottom row shows that to avoid the fingers\footnote{Snapshots extracted from the video: \url{https://www.youtube.com/watch?v=Iuj3oYgTILQ}} coming into contact with the surface, the gripper must be properly positioned with a vertical margin.
	\end{comment}

	As a consequence, when performing top graspings by teleoperation, the fingertips may collide with the surface giving rise to unnecessary normal forces due to the difficulty in continuously adjusting the hand position or finding the appropriate position of the hand midair.
	{\color{black}
		Empirically, we found that when using the original HSR gripper, normal contact  forces beyond the range 50$\sim$60 N---sensed by the original HSR force-torque sensor mounted on the wrist---would at times raise an emergency stop signal, rendering the robot unresponsive.
		As such, within the scope of this paper, we consider delicate graspings to be those that generate  contact forces of less than 40 N.
	}

	\begin{figure}
		\centering
		\vspace{.2cm}
		\includegraphics[width=0.999\linewidth]{hsr_f1_z_height_d2_compressed}
		\caption{
			Top row: when grasping flat and wide objects on a hard surface, a teleoperator has to carefully move the hand upwards, as the fingers are closed.
			Bottom row: the hand must be placed at an appropriate height such that the finger does not hit the table surface before it makes contact with the object.
		}
		\label{fig:hsr_f1_z_height}
	\end{figure}

	\subsection{Fixed-Finger Grippers}

	\color{black}
	In general, a solution to decrease a mechanism's cost and increase its reliability is by simplifying the  design and reducing the number of parts required.
	For a two-finger gripper, this can be achieved by eliminating actuation in one of the fingers.
	While this loss of motion may require atypical grasping approaches, one upside is that switching and designing tailored fixed-fingers with different structures is much easier than for an actuated finger.
	\color{black}

	For example, ugo, a security robot, has this type of gripper. However, its grippers are mainly used to press elevator buttons and rarely to hold objects \footnote{Demonstration of ugo: {\color{black}\url{https://www.youtube.com/watch?v=9xybTXT9hYU}}}.
	Pastor et al. use this gripper to ensure a stable and large surface for mounting tactile sensors \cite{pastorUsing3dConvolutional2019}. The setup was used to investigate in-hand manipulation. In particular, palpatation movements due to the actuated finger were used to recognize objects using a deep neural network.

	\begin{comment}
	Apart from the usual grippers found in commercial robots, the literature on gripper design is extremely vast and covers a wide range of ingenious designs.
	Even if we constrain the literature to only two-finger designs, the body of work can range from soft grippers with multi-segments \cite{teeple2020_hand} to designs specific for thin \cite{yoshimi2012_hand} and  small \cite{watanabe2021_hand} objects.
	Here, we narrow our scope and focus on a class of much less explored two-finger grippers where one of the fingers is fixed.

	The body of work of fixed-fingers is limited and there seems to be no unique reason for its adoption.
	The robot Ugo from Mira Robotics has a bi-manual configuration with fixed fingers.
	Although a demonstration of the robot folding laundry under teleoperation is available as a video\footnote{\url{https://www.youtube.com/watch?v=iZn_vrQdkYQ}}, the company has not formally published or technically evaluated the gripper design.
	In this case, the fixed finger appears to be used for design simplicity.
	In \cite{pastorUsing3dConvolutional2019}, a fixed finger was used as a stable, large surface for the attachment of a tactile sensor.
	The hand is shown mounted on an arm, but the grasping of objects initially out of the hand was not studied.
	Rather, the setup was used to investigate in-hand manipulation.
	In particular, palpatation movements due to the actuated finger were used to recognize objects using a deep neural network.

	\end{comment}

	Inspired by surgical palpatation, the M$^2$ gripper \cite{maM2GripperExtending2016} is a gripper whose design is aimed at dexterous in-hand manipulation where the fixed finger acts as a stable thumb counter-part of the forefinger.
	To achieve dexterities such as in-hand controlled slip, two independent  actuators are required to allow for the control of the agonist and antagonist tendons of the forefinger.
	A similar design uses an active surface in the form of a belt on the fixed finger \cite{maInhandManipulationPrimitives2016}.
	In both cases, the grippers were evaluated with a static base---not as part of a grasping robot system---and at times with a fixed arm motion as their focus was on in-hand manipulation.
	The gripper was used in \cite{zhangDoraPickerAutonomousPicking2016} where experiments indicate that the capabilities of a simple vacuum gripper and a symmetrically actuated gripper are complimentary to the M$^2$ when grasping a wide range of objects.

	\begin{comment}

	\subsection{Contribution}
	Our contribution is twofold.
	First, we introduce the F1 hand, a \emph{novel fixed-finger hand} that is mechanically simple, low cost, and allows for great versatility as the easy-to-switch fixed finger can be manufactured in a variety of shapes (Fig. \ref{fig:intro}(a)).
	The design allows for a constant grasp height during the closing of the actuated finger, thus overcoming the problem introduced in Fig. \ref{fig:hsr_f1_z_height}.
	The hand can adaptively change the grasping configuration according to the object's geometry, and thus grasp a variety of objects from a small paperclip to a bulky hand drill.

	A second contribution addresses the control challenges introduced by the atypical asymmetric design of the F1 hand.
	To this end, we propose a \emph{grasping controller} that uses the whole arm/robot movements to compensate for the asymmetric closing of the gripper.

	We show that the grasping controller allows us to use the hand as if it was symmetrically actuated by directly feeding the results of an off-the-shelf method for grasp pose estimation.

	\end{comment}

	\section{The F1 Hand Design}

	The proposed F1 hand uses a fixed finger for three main purposes.
	First, the fixed finger provides a physical and fixed end-effector tip that dictates the constant grasp height between the object and the hand as shown in Fig. \ref{fig:F1_advantages}(a) and is particularly useful to facilitate delicate top grasp teleoperation.
	Second, as previously motivated, the ability to freely shape and change the configuration of the hand\footnote{A promotional video of the hand with different attachments can be watched here: \url{https://www.youtube.com/watch?v=sN66DilliBs}}.
	Third, as shown in Fig.\ref{fig:F1_advantages}(b) the fixed finger acts as the supporting counterpart of the multi-link actuated movable finger, allowing for adaptive enveloping or pinch grasps depending on the shape of the object.
	The hand combines the fixed-height grasping benefits of a parallel gripper with the advantages of the adaptive finger for delicate and robust grasps.

	\begin{figure}
		\centering
		\vspace{0.3cm}
		\includegraphics[width=0.99\linewidth]{F1_advantages_d_compressed}
		\caption{
			(a) The fixed finger provides a visual guide to check the distance to the object that also dictates a constant grasp height which is particularly useful in teleoperation.
			(b) The multi-linkage finger allows a continuum of adaptive grasps varying from pinch to enveloping grasps that only depends on the positions of the grasping interaction forces.
		}
		\label{fig:F1_advantages}
	\end{figure}

	Fig.~\ref{fig:f1wholedesign} shows the structure and dimensions of the hand.
	The fixed finger is attached to a vertical linear sliding holder which is maintained in its initial extended state by a push spring.
	The slider allows the fixed finger to retract inside the palm under contact with objects or surfaces.
	%

	%

	{\color{black}
		This retraction effectively creates a margin for positioning error.
		Not only the spring allows for the compensation of small errors in the approaching direction of the grasping, thus increasing success rates, but also protects the hand from damage.
	}
	\begin{figure}
		\centering
		\includegraphics[width=0.85\linewidth]{f1_whole_stB01}
		\caption{Structure {\color{black}of F1 hand having 	a fixed finger with} a sliding mechanism.}
		\label{fig:f1wholedesign}
	\end{figure}

	As shown in Fig. \ref{fig:f1fingerdesign}(a), the actuated finger is a multiple linkage mechanism based on a hand developed by one of the authors in the past~\cite{fukaya2000_hand, fukaya2013_hand}, but here constructed compactly due to a modification of the link B. The size was determined based on the middle finger of
	{\color{black} an adult male}. The three joints (MP joint, PIP joint, and DIP joint) can be operated in conjunction by applying a driving force to one point.
	Two torsion springs are attached to the PIP joint of the movable finger to maintain the finger in extension as the basic posture, and the spring constant of the torsion spring is K=0.13 N$\cdot$mm/deg.

	The actuated finger is driven by a motor (Dynamixel XM430-W350-R).
	The motor and the movable finger are connected by the Link A to transmit the driving force.
	The driving state of the finger differs depending on which part of the finger contacts the object and allows for highly adaptive grasps.
	As shown in Fig.~\ref{fig:f1fingerdesign}(b), when the distal phalanx link contacts the object, the movable finger transmits the grasping force while maintaining the extended state.
	If the position of the object is shifted when the finger contacts the object, the finger automatically changes to a posture that allows stable contact.

	Conversely, as shown in Fig.~\ref{fig:f1fingerdesign}(c), if the proximal phalanx link touches the object, the link D moves independently and actuates the middle phalanx via link B, while the link C (which is connected to the proximal phalanx link) pulls the distal phalanx link.
	As a result, the movable finger moves such that its segments wrap around the object, and the grasping motion automatically stops when all possible contact parts are touched.
	As the contact area is maximized, the contact forces of each part is distributed and the object can be grasped stably even with small motor torques.

	\begin{figure}
		\centering
		\vspace{0.2cm}
		\includegraphics[width=0.9\linewidth]{f1_finger_st_B01b}
		\vspace{-0.2cm}
		\caption{{\color{black}Illustration} of the design and motion of the movable finger.
			(a) Kinematic structure.
			(b) Final configuration under fingertip contact.
			(c) Final configuration under even contact where all three links envelop the object.
		}
		\label{fig:f1fingerdesign}
	\end{figure}

	\section{Grasping with the F1 Hand}

	When naively grasping small or thin objects with the F1 hand, one needs to carefully reason the reachability of the fingers.
	As shown in the {\color{black}left illustration} of Fig. \ref{fig:controlstrategy}(a), if the fixed finger is too far from the object, closing the gripper with the actuated finger while maintaining the hand static would eventually lead to the uncontrolled sliding or rolling of the object;
	{\color{black} which among other factors, depends on the unknown friction between the parts.}
	{\color{black}
		As a measure, one could position the fixed finger as close as possible to the object before executing the grasp as shown in the right case in Fig. \ref{fig:controlstrategy}(a).
		However, during the pre-grasping motion, this poses the risk of the fixed finger accidentally hitting the object, disturbing it from its original position.
	}

	\subsection{Grasping in Teleoperation}    \label{sec:teleop_controller}

	\begin{figure}
		\centering
		\includegraphics[width=0.9997\linewidth]{control_strategy}
		\caption{ {\color{black} (a) The fixed finger is too far from the object when the grasping starts and the actuated finger eventually rolls or slides the object (left), potentially leading to grasping failures.
				Starting the grasp close to the object can be difficult due to the required positioning requirements and possible disturbance on the initial object position (right).
				(b) A natural approach is to assume that the object lies somewhere in between the current aperture of the hand $D$.
		}}
		\label{fig:controlstrategy}
	\end{figure}

	Fig. \ref{fig:controlstrategy}(b) illustrates a strategy that resembles that of a conventional two-finger gripper.
	It assumes that the object is centered between the two fingers' projected distance on the grasping surface, indicated by the length $D$.
	To grasp, the object is approached by both fingers simultaneously.
	This is done with the use of a grasping primitive that closes the actuated finger with a motor command $q$, while moving the entire hand (and thus robot) in the direction $u$ over a distance of $0.5 D$.

	The robot control is accomplished by moving the $O_{tip}$ horizontally as a Cartesian target towards the center of the object and solving the inverse kinematics for the whole robot motion.
	In the same control loop, incremental commands for the actuated finger are sent such that by the end of the process the actuated finger is closed.
	As the robot moves and the actuated finger closes, contact will be eventually made, resulting in the grasping of the object.
	If the object is perfectly centered, both fingers will meet the left and right walls of the object at the same time.
	%

	{\color{black}
		In practice, the level of accuracy in which an operator can center the object between the fingers highly depends on his/her experience with the system. However, in general, we noticed that the approach of using a grasping primitive of Fig. \ref{fig:controlstrategy}(b) is much more forgiving in terms of positioning error when compared to the strategies shown in Fig. \ref{fig:controlstrategy}(a).
	}

	\subsection{Autonomous Grasping} \label{sec:autonomous_grasping}

	\begin{figure}
		\centering
		\vspace{0.2cm}
		\includegraphics[width=0.95\linewidth]{control_strategy_pose_estimator}
		\vspace{-0.2cm}
		\caption{
			Parametrization of the grasping controller based on the values output by an off-the-shelf grasp pose estimator GG-CNN \cite{morrisonLearningRobustRealtime2020} (the right figure is adapted from the same paper).
		}
		\label{fig:controlstrategyposeestimator}
	\end{figure}

	\begin{figure}
		\centering
		\includegraphics[width=0.9\linewidth]{experimental_setup_2}
		\vspace{-0.1cm}
		\caption{
			Experimental setup. (a) The 3D mouse interface and {\color{black}the two HSRs used for experiments where the left HSR equipped with F1 hand shows the teleoperation initial position}.
			(b) The thin fixed finger {\color{black}(upper)} was used for teleoperation experiments and the thicker version {\color{black}(lower)} was used for autonomous grasping experiments.
		}
		\label{fig:robot_setup}
	\end{figure}

	\begin{figure*}
		\centering
		\vspace{0.2cm}
		\includegraphics[width=0.95\linewidth]{objects2}
		\vspace{-0.2cm}
		\caption{
			The objects used for experiments are based on a subset of YCB objects with the addition of the toothpick, the coin, and the paper clip.
			The pose of each object in the picture also indicates how they were placed on the table during top graspings.
			The numbers are used to identify each of the objects.
			Teleoperation experiments can be watched in the first part of the video here: \href{https://youtu.be/iWXXIX4Mkl8}{https://youtu.be/iWXXIX4Mkl8}
		}
		\label{fig:objects}
	\end{figure*}

	As shown in Fig. \ref{fig:controlstrategyposeestimator}, we assume that the robot is provided with an approaching direction as vector $v$ and that there are two points of contact (one for each finger) symmetrically opposed on a plane containing $v$ and separated by a distance $w$.
	This assumption is in accordance with many of the recent vision-based grasp pose estimators found in the literature, including 6-DoF estimators (e.g. \cite{mahlerDexnetDeepLearning2017, murali20196, morrisonLearningRobustRealtime2020, breyerVolumetricGraspingNetwork2021}).
	In the particular case of planar top grasps, $v$ is normal to the surface to which the camera point towards.
	As the $O_{tip}$ of the F1 hand is defined to be at the tip of the fixed finger we set its pre-grasping position to be $d=w/2$ and open the actuated finger to $w$.
	We then call the same primitive grasping controller used for teleoperation.

	The described process provides the parameters of the grasping primitive in which the input is the center of the object to be grasped.
	If a grasp pose estimator also provides the gripper aperture $w$, the value of $d$ and the initial angle of the actuated finger can be specified.
	This allows us to treat the hand as if it was a parallel symmetrical gripper as shown in
	Fig. \ref{fig:controlstrategyposeestimator} where the controller maps the output of the grasp pose estimator GG-CNN \cite{morrisonLearningRobustRealtime2020} to the asymmetric closing of the gripper.

	\section{Experiments} \label{sec:experiments}

	Two identical Toyota HSRs were used, one with the original gripper and another one with the F1 hand, as shown in Fig. \ref{fig:robot_setup}(a).
	The F1 hand's motor, the 3D mouse, and the RealSense D435i were connected via USB cable.
	The HSR was connected to a desktop PC via LAN and running ROS.
	In teleoperation, the operator sitting in front of the robot used a Connexion 3D mouse to move the reference position of the end-effector $O_{tip}$.
	To make a fair comparison with the original HSR gripper, we fixed the shape of the fixed finger to a narrow design (left picture in Fig. \ref{fig:robot_setup}(b)) and did not change it during the teleoperation experiments.
	{\color{black}
		The contact forces were measured using the original force-torque sensors of the HSR, located at the flange part of the gripper mount.
		The F1 hand and its mounting point were designed such that the attachment of the F1 hand did not affect the functionality of the force-torque sensor.
	}

	In autonomous grasping, we implemented the  GG-CNN~\cite{morrisonLearningRobustRealtime2020} as a popular top grasp pose estimator and mapped its output parameters to the grasping controller.
	The GG-CNN used the RealSense D435i camera attached to the back of the hand of the robot.
	We used a wider version of the fixed finger to increase the robustness to inaccurate pose estimations and calibration errors (right finger in Fig. \ref{fig:robot_setup}(b)).
	The experiments can be watched by following the link: \href{https://youtu.be/iWXXIX4Mkl8}{https://youtu.be/iWXXIX4Mkl8}

	\subsection{Experiments in Teleoperation}

	As shown in Fig. \ref{fig:objects}, 22 objects from the YCB dataset  \cite{calliYaleCMUBerkeleyDatasetRobotic2017a} were selected and were added with three additional smaller and thinner objects: a Japanese one-Yen coin (diameter 20 mm, thickness 1.5 mm), a small clip (length 27 mm, thickness 0.7 mm), and a toothpick (length 60 mm, diameter 2 mm).
	Together with the YCB small washer (diameter 10 mm, thickness 1.2 mm), these additional set of objects become extremely challenging for general-purpose grippers.

	Although many authors have proposed grippers that can grasp small objects (e.g. \cite{yoshimi2012_hand, watanabe2021_hand}), such designs render the gripper  tailored to a specific range of sizes.
	Conversely, here we use the YCB subset in conjunction with our added small objects to provide evidence that our hand can cover a wide variety of shapes, sizes, and weights.
	The subset of 22 objects, shown in  Fig. \ref{fig:objects}, were selected based on their shape and weight.

	We favored objects with curvature, excluding the many package boxes, Lego blocks, and wooden blocks due to their trivial grasping geometry.
	The mug (\#8) and the Pringles can (\#14) were selected as representative of the many cups and other cylindrical objects of equivalent size.
	Plastic containers with complex curvatures are represented by the mustard and the Windex bottle (\#23, \#24).
	The tennis ball (\#19) and the plastic strawberry (\#20) were selected to represent the numerous rounded plastic objects shaped as fruits.
	Among the many washers, screws, and nuts of the dataset, we selected a small washer of the dataset (\#3) and a screw (\#5).
	The electric drill (\#16) and the toy airplane (\#21) were selected due to their large size and weight.

	Our selection of objects covered the following ranges.
	The heaviest object was the hand drill (600 g).
	The thinnest object was the paperclip  (0.7 mm).
	The smallest object was the washer ($\phi$10 mm $\times$ 1.2 mm)
	The biggest object was the toy airplane (bounding box of approximately 26 cm $\times$ 28 cm $\times$ 20 cm).
	Other notable objects that were not included in the experiments are discussed in Sec.  \ref{sec:limitation_and_extension}.

	As one of the most basic and frequently used motions in teleoperation, we limited the scope of experiments to top grasps only.
	As such, only the yaw angle $\theta$ was controlled together with the ($x,y,z$) directions (see Fig. \ref{fig:robot_setup}).
	%

	{\color{black}
		For each object, 20 to 21 teleoperation trials were executed by an expert operator whose goal was to grasp as fast as possible.
		To the best of our efforts, we believe the effects of learning curve and biases towards one of the gripper designs were not present.
		The teleoperator was used to the 3D mouse interface and we assured he had been practicing graspings for a few weeks with both the original HSR gripper and the F1 hands before the experiments were recorded.
	}

	The trials were randomized by fixing the object and robot's initial poses while randomly setting the initial rotation of the wrist by sampling the rotation angle from a uniform distribution in the range $[-45^{\circ}, 45^{\circ}]$.
	The random initial rotation forced the operator to teleoperate the robot to bring the gripper to a pre-grasp position.
	Once the pre-grasping position was achieved, the operator pressed a button that in the case of the HSR gripper activated the torque-controlled grasp, and in the case of F1 hands executed the primitive described in {\color{black}Sec. \ref{sec:teleop_controller}}.
	The robot then lifted the object 10 cm such that the operator could visually confirm the grasp success.

	Figure \ref{fig:result_teleop}(a) summarizes the success rate individually.
	The success rates of the original HSR gripper for the coin (\#1) and the washer (\#3) were zero.
	Despite our best efforts, the original HSR gripper could not appropriately grasp these two objects.
	In many trials, the torque controller to close the gripper would fail as the tips of the fingers would touch the surface before grasping the object, eventually leading the robot unresponsive as the fingers could not be closed and the gripper could not be retracted due to the ROS blocking service.
	The aggregated success rate including all objects was 89\% for the original HSR gripper and increased to 94\% when using the F1 hands.

	Figure \ref{fig:result_teleop}(b) shows the mean (bar height) and one standard deviation (circle) of the 20 trials in terms of the peak normal contact force and the grasping time while (c) shows the histogram over all trials and all objects (460 runs in each robot).
	The F1 hand showed lower contact force with the surfaces over all object cases.
	The mode of the histogram of the F1 hand is 4.28 N while {\color{black}that of the original HSR gripper} is 18.78 N, representing a reduction of 77\% in peak force.
	Teleoperation time decreased in average by 3 seconds which makes grasping with the F1 hands 20\% faster.
	For the F1 hands, the time histogram has a mean $\pm$ one standard deviation value of 10.99 $\pm$ 4.06 seconds while {\color{black} that of the original HSR gripper} is  13.88 $\pm$ 5.83 seconds.

	{\color{black}
		We also examine the performance with the another expert operator, and found the results are quite consistent.
		The overall success rates, contact force, and grasping time averaged among the objects are 88\%, 21.43 N, and 13.49  $\pm$ 2.1 seconds for original HSR gripper and 95\%, 4.32 N, 11.2 $\pm$ 1.65 seconds, for F1 hand, respectively.}

	\begin{figure}
		\centering
		\vspace{.2cm}
		\includegraphics[width=.94\linewidth]{result_individual_and_aggregated}
		\vspace{-.002cm}
		\caption{
			Results of the teleoperation where each of the 25 objects were grasped between 20 to 21 times.
			(a) The aggregated success rates over all objects of the F1 and the original HSR gripper.
			(b) Individual peak contact force and time to grasp.
			(c) The histogram of the 460 grasps, excluding the objects \#1 and \#3 for which the HSR data is not available.
		}
		\label{fig:result_teleop}
	\end{figure}

	\begin{figure*}
		\centering
		\vspace{0.2cm}
		\includegraphics[width=0.99\linewidth]{snapshot_grasping_pen_ggcnn2}
		\vspace{-0.2cm}
		\caption{
			A sequence of snapshots of the F1 hand grasping a pen in autonomous mode.
			The grasping pose was estimated with the GG-CNN.
			The entire action can be divided in two segments.
			In the first, the robot positions $O_{tip}$ as a pre-grasping motion.
			In the second, the robot executes the primitive control as it was described in Sec. \ref{sec:autonomous_grasping}.
		}
		\label{fig:snapshotgraspingpenggcnn}
	\end{figure*}

	{\color{black}
		Despite both robots being identical, the decrease in grasping time with the F1 hand was visible.
		To its advantage, the F1 hand does not require adjusting the height of the gripper while the fingers close, as is the case with the original HSR gripper, which saves time.
		Also, the primitive controller allows for a rough initial positioning of the hand as the adaptive movable finger assimilates the shape of the object while using the fixed finger as a supporting counter-part.
		As a result, the margin for relative error in the positioning of the F1 hand is larger than when grasping objects with the HSR gripper which leads to faster operations.
	}

	\subsection{Autonomous Grasping}

	Due to its popularity and simplicity, the GG-CNN~\cite{morrisonLearningRobustRealtime2020} was used to investigate the efficacy of the primitive described in Sec. \ref{sec:autonomous_grasping} in autonomous mode.
	We selected ten of the objects used in the teleoperation dataset.
	We eliminated the small objects (\#1 to \#5) due to a lack of resolution in depth perception  of the RealSense D435i.
	For the bigger objects, we selected representative objects that lead to similar  pose estimation by the GG-CNN.
	As such, for the objects with a rim (\#8, \#12, \#17, \#22) we selected the \#17 as the most challenging one.
	For cylindrical objects (\#9, \#14, \#18, \#25) we used \#18  and \#25.
	For the two spherical objects (\#19, \#20) we selected \#20.
	We used the toy drill \#15 instead of the real one \#16.
	The airplane \#21 and the mustard \#23 were excluded due to their height because often the camera would be hit during the approaching phase or due to extraneous grasping poses\footnote{We acknowledge this problem can be solved by using a motion planner on the point cloud during pre-grasping and we leave such addition  as future work.}, which would require stopping the experiments, assessing, and eventually re-doing calibration procedures.

	Each object was placed individually and centered on a table in front of the robot and 20 trials were executed.
	The initial object position was changed by rotating approximately 30$^\circ \sim$ 45$^\circ$ from the previous trial.
	The robot started by going to a home position where the camera on the wrist was used to capture the RGB-D image from a top-down configuration.
	Under this pose, the GG-CNN was used to infer the ($x,y,z$) and the $\theta$ parameter of the grasping pose.

	Once the grasp was detected, the controller mapped the $(x,y,z, \theta)$  values to the grasping primitive shown in Fig. \ref{fig:controlstrategyposeestimator}.
	As shown in Fig. \ref{fig:snapshotgraspingpenggcnn}, the robot used a straight trajectory as a pre-grasping approach to move from its current position to the start of the primitive motion, before executing the grasping controller.
	Then, the robot attempted to lift the object 10 cm above the table (Fig. \ref{fig:testshot_ggcnn}).
	Human supervision was used to verify grasp success.
	To account for inaccuracies of the pose estimator and calibration errors, we used a wider version of the fixed-finger gripper (right finger in Fig. \ref{fig:robot_setup}(b)).

	\begin{figure}
		\centering
		\includegraphics[width=0.97\linewidth]{testshot_of_GGCNN_a}
		\vspace{-0.2cm}
		\caption{
			Experimental example of autonomous grasping using GGCNN.
			The red {\color{black}rectangles} indicate the grasping points estimated by the GGCNN.
			During operation, the robot first aligns the gripper orientation before starting the execution of the grasping. The adaptive mechanism increases the success rate of grasping even for hard-to-grasp objects such as the scissors on the right.
			Autonomous grasping experiments can be watched in the second part of the video here: \href{https://youtu.be/iWXXIX4Mkl8}{https://youtu.be/iWXXIX4Mkl8}
		}
		\color{black}
		\label{fig:testshot_ggcnn}
	\end{figure}

	The success rate with both hands shows no significant difference as indicated by Fig. \ref{fig:successrate_ggcnn} where the F1 hand and the HSR original gripper achieved an overall score of 0.84 and 0.82, respectively.
	Similar trends are observed on a per-object basis, an indication that the success disparity among objects is mainly due to the adequacy of the grasp pose estimation rather than the hand capabilities.
	Indeed, for certain object orientations, the estimator would estimate extraneous grasping orientations such as grasping a pen with the finger aligned with the pen's length, which led to failure regardless of the gripper in use.

	\begin{figure}
		\centering
		\includegraphics[width=0.97\linewidth]{success_rate}
		\vspace{-0.2cm}
		\caption{
			Success rates of autonomous grasping trials with both the original HSR gripper and the F1 hand.
			A similar trend was observed per object due to erratic and/or suboptimal grasp pose estimations from the GG-CNN which affected both grippers equally.
			The objects and their indices are indicated in Fig. \ref{fig:objects}.
		}
		\label{fig:successrate_ggcnn}
	\end{figure}

	This experiment's result allows us to verify that our system could successfully use the output of a grasp pose estimator that assumes symmetric actuation by leveraging the whole-body motion of the robot with the use of the grasping primitive actions.
	The fact that similar success rates were achieved by the HSR original gripper, indicates that the  mapping by the controller did not decrease the success rate.

	\begin{figure*}
		\centering
		\vspace{0.2cm}
		\includegraphics[width=0.9850\linewidth]{other_finger_designs}
		\vspace{-0.2cm}
		\caption{
			Challenging YCB objects.
			(a-d) A fixed finger with a latch mechanism makes it possible to slide the fixed finger on a linear guide to increase the hand aperture.
			(e) Grasping the plate and the table cloth are possible when enabling the roll motion of the hand.
			(f) Grasping the frying pan is still challenging due to the large momentum exerted on the wrist.
		}
		\label{fig:otherfingerdesigns}
	\end{figure*}

	\section{Extending Grasping to other YCB Objects} \label{sec:limitation_and_extension}

	We discuss some of the most challenging YCB objects not included in the experiments due to the current limitations of the system and how we plan to address them.
	Using the fingers of Sec. \ref{sec:experiments} the F1 hand was not capable of grasping some of the largest objects of the dataset.
	Namely, the mini-soccer ball, the wooden block, and the jar.
	Currently, we are investigating a sliding fixed finger that uses a linear guide and a latch to allow for quick and easy change of the hand opening.
	This functionality increases the distance between fingers from 130.5 mm to 215 mm and makes it possible  to grasp not only bigger objects not previously possible---shown in Figs. \ref{fig:otherfingerdesigns}(a-c)---but also smaller objects such as a pen  as shown in (d).

	The plate and the table cloths are graspable but they require approaching the objects sideways, which was out of the scope of our top grasp study. Finally, the issue of the frying pan is having the position of its heavy center of mass (0.8 kg) far away from the handle and thus causing unstable grasps due to excessive momentum on the wrist.
	This could be solved with wide fixed finger designs that could act as a wide palm to provide support to the large momentum and remain as one of the challenges for our next iterations of hand design.

	\section{CONCLUSIONS}

	We introduced the F1 hand, a novel versatile two-fingered gripper with a fixed finger, and evaluated it as part of a robotic system.
	This fixed finger facilitates the visual assessment of the relative position of the hand to the object during the grasping operation, thus enabling a more intuitive teleoperation.
	A controller was introduced to preserve the intuitive approach of centering objects between the fingers by mapping symmetry parameters to the asymmetric actuation of the hand.
	In total, together with teleoperation and autonomous graspings, approximately 700 grasps were executed with the F1 hands mounted on a Toyota HSR using the proposed control method.
	Compared to the original HSR gripper, under teleoperation, our hand provided faster grasping while generating significantly less force on the surfaces, with a slight increase in success rates. Also, when using an off-the-shelf grasp pose estimator, the autonomous grasping success rate was comparable to the original HSR gripper.

	\section*{ACKNOWLEDGMENT}

	The authors cordially express their gratitude to Mr. Shimpei Masuda, Dr. Tianyi Ko and Dr. Koji Terada in Preferred Networks, Inc. and Mr. Koichi Ikeda, Mr. Hiroshi Bito and Dr. Hideki Kajima in Toyota Motor Corporation for their assistance.