HOnnotate: A method for 3D Annotation of Hand and Object Poses

Shreyas Hampali1, Mahdi Rad1, Markus Oberweger1, and Vincent Lepetit2,1

1Institute for Computer Graphics and Vision, Graz University of Technology, Austria2LIGM, Ecole des Ponts, Univ Gustave Eiffel, CNRS, Marne-la-Valle, France

hampali, rad, oberweger, lepetit@icg.tugraz.at, Project Page: https://www.tugraz.at/index.php?id=40231

Abstract

We propose a method for annotating images of a handmanipulating an object with the 3D poses of both the handand the object, together with a dataset created using thismethod. Our motivation is the current lack of annotatedreal images for this problem, as estimating the 3D poses ischallenging, mostly because of the mutual occlusions be-tween the hand and the object. To tackle this challenge, wecapture sequences with one or several RGB-D cameras andjointly optimize the 3D hand and object poses over all theframes simultaneously. This method allows us to automat-ically annotate each frame with accurate estimates of theposes, despite large mutual occlusions. With this method,we created HO-3D, the first markerless dataset of color im-ages with 3D annotations for both the hand and object. Thisdataset is currently made of 77,558 frames, 68 sequences,10 persons, and 10 objects. Using our dataset, we developa single RGB image-based method to predict the hand posewhen interacting with objects under severe occlusions andshow it generalizes to objects not seen in the dataset.

1. IntroductionMethods for 3D pose estimation of rigid objects and

hands from monocular images have made significantprogress recently, thanks to the development of Deep Learn-ing, and the creation of large datasets or the use of syn-thetic images for training [33, 36, 47, 57, 73, 75]. However,these recent methods still fail when a hand interacts with anobject, mostly because of large mutual occlusions, and ofthe absence of datasets specific to 3D pose estimation forhand+object interaction. Breaking this limit is highly de-sirable though, as 3D hand and object poses would be veryuseful in augmented reality applications, or for learning-by-imitation in robotics, for example.

Several pioneer works have already considered this prob-lem, sometimes with impressive success [27, 54, 63]. Theseworks typically rely on tracking algorithms to exploit

temporal constraints, often also considering physical con-straints between the hand and the object to improve the poseestimates. While these temporal and physical constraints re-main relevant, we would like to also benefit from the powerof data-driven methods for 3D hand+object pose estimationfrom a single image. Being able to estimate these posesfrom a single frame would avoid manual initialization anddrift of tracking algorithms. A data-driven approach, how-ever, requires real or synthetic images annotated with the3D poses of the object and the hand. Unfortunately, cre-ating annotated data for the hand+object problem is verychallenging. Both common options for creating 3D annota-tions, annotating real images and generating synthetic im-ages, raise challenging problems.

Annotating real images. One can rely on some algo-rithm for automated annotation, as was done for currentbenchmarks in 3D hand pose estimation [46, 56, 59, 73, 76],where the “ground truth” annotations are obtained automat-ically with a tracking algorithm. Though these annotationsare noisy, they are usually taken for granted and used fortraining and evaluation [38]. Another approach is to usesensors attached to the hand as in [15] (bottom right imageof Fig. 1). This directly provides the 3D poses, however,the sensors are visible in the images, and thus bias learningmethods. Significant effort is still required in developingalgorithms for automated annotation of real images.

Generating synthetic images. Relying on syntheticimages is appealing, as the 3D poses are known perfectly.Realistic rendering and domain transfer can be used to train3D pose estimation on synthetic images [32, 48, 75]. Gen-erating physically correct grasps is possible [30], as shownin [19], but complex manipulation is difficult to simulate.However, real images with accurate 3D annotations wouldstill be needed to evaluate the generalizability of the methodto real data.

We therefore propose a method to automatically annotatereal images of hands grasping objects with their 3D poses.Our method works with a single RGB-D camera, but canexploit more cameras if available for better robustness and

1

arX

iv:1

907.

0148

1v6

[cs

.CV

] 3

0 M

ay 2

020

[76] [54]

[19] [15]Our proposed HO-3D dataset Existing datasets

Figure 1: We introduce a method for labelling real images of hand+object interaction with the 3D poses of the hand and ofthe object. With this method, we automatically created a dataset made of more than 75,000 frames, 10 different objects and10 different users. In comparison, existing datasets have several limitations: The 3D objects are very simple, the interactionis not realistic, the images are synthetic, corrupted by sensors, and/or the number of samples is limited. More illustrations ofannotations in our dataset are shown in the supplementary material.

accuracy. The single-camera setup works under the assump-tion that the grasp pose varies marginally over the sequence;the multi-camera setup can handle complex hand+object in-teraction scenarios. Instead of tracking the poses frame-by-frame, our method optimizes jointly all the 3D poses of thehand and the object over the sequence. As our evaluationsshow, this allows us to exploit temporal consistency in astronger way than a tracking algorithm. Using differentiablerendering, we can optimize a complex objective function byexploiting the new powerful gradient descent methods orig-inally developed for Deep Learning [25]. We see this ap-proach as the equivalent of bundle adjustment for SLAMalgorithms, where we track objects instead of points.

We rely on the MANO hand model [51], and the 3Dmodel of the objects. We use objects from the YCB-Videodataset [70], as they have various shapes and materials, andcan be bought online [1] by researchers interested in per-forming their own experiments. Being able to use a sin-gle camera also enables easier expansion of the dataset byother researchers with a larger variety of objects and grasp-ing poses as multi-camera capture is often complex to setup.

Using our method, we created a dataset, depicted inFig. 1, which we call HO-3D. In addition, we used thisdataset to learn to predict from a single RGB image the 3Dpose of a hand manipulating an object. More exactly, wetrain a Deep Network to predict the 2D joint locations ofthe hand along with the joint direction vectors and lift themto 3D by fitting a MANO model to these predictions. Thisvalidates the fact that the 3D poses estimated by our anno-tation method can actually be used in a data-driven methodfor hand pose estimation. By comparing with an existingmethod for hand+object pose estimation [19] that directlyestimates MANO parameters, we show that predicting 2Dkeypoints and lifting them to 3D performs more accurately.

2. Related Work

The literature on hand and object pose estimation is ex-tremely broad, and we review only some works here.

2.1. 3D Object Pose Estimation

Estimating the 3D pose of an object from a single frameis still one of the fundamental problems of Computer Vi-sion. Some methods are now robust to partial occlu-sions [21, 37, 42], but many works rely on RGB-D data tohandle this problem [5, 8, 23, 31], by fitting the 3D objectmodel to depth data. This can fail when a hand grasps theobject, since the surface of the hand can be mistaken for thesurface of the object.

2.2. 3D Hand Pose Estimation

Single image hand pose estimation is also a very pop-ular problem in Computer Vision, and approaches can bedivided into discriminative and generative methods. Dis-criminative approaches directly predict the joint locationsfrom RGB or RGB-D images. Recent works based on DeepNetworks [16,33,35,36,61,71,75] show remarkable perfor-mance, compared to early works based on Random Forestssuch as [24]. However, discriminative methods performpoorly in case of partial occlusion.

Generative approaches take advantage of a hand modeland its kinematic structure to generate hand pose hypothe-ses that are physically plausible [13, 29, 46, 52, 55, 65, 72].[32, 41] predict 2D joint locations and then lift them to3D. Generative approaches are usually accurate and can bemade robust to partial occlusions. They typically rely onsome pose prior, which may require manual initialization orresult in drift when tracking.

Our work is related to both discriminative and generativeapproaches: we use a generative approach within a globaloptimization framework to generate the pose annotations,

and use a discriminative method to initialize this complexoptimization. [66] also combines generative and discrim-inative approaches to train a network in a self-supervisedsetting. However, they only consider hands. We also train adiscriminative method using our dataset, to predict the handposes which are robust to occlusions from interacting ob-jects.

2.3. Synthetic Images for 3D Pose Estimation

Being able to train discriminative methods on syntheticdata is valuable as it is difficult to acquire annotations forreal images [75]. [19, 48] show that because of the domaingap between synthetic and real images, training on syntheticimages only results in sub-optimal performance. A GANmethod was used in [32] to make synthetic images of handsmore realistic. While using synthetic images remains ap-pealing for many problems, creating the virtual scenes canbe expensive and time-consuming. Generating animated re-alistic hand grasps of various objects, as it would be re-quired to solve the problem considered in this paper remainschallenging. Being able to use real sequences for traininghas thus also its advantages. Moreover, evaluation shouldbe performed on real images.

2.4. Joint Hand+Object Pose Estimation

Early approaches for joint hand+object pose estima-tion [2,39,67] typically relied on multi-view camera setups,and frame-by-frame tracking methods, which may requirecareful initialization and drift over time. [40, 64] proposegenerative methods to track finger contact points for in-handRGB-D object shape scanning. [44, 45] consider sensingfrom vision to estimate contact forces during hand+objectinteractions using a single RGB-D camera, and then esti-mate the hand and the object pose. However, these methodsare limited to small occlusions.

[27,63] propose to use a physics simulator and a 3D ren-derer for frame-to-frame tracking of hands and objects fromRGB-D. [28] uses an ensemble of Collaborative Trackersfor multi-object and multiple hand tracking from RGB-D images. The accuracy of these methods seems to bequalitatively high, but as the ground truth acquisition in areal-world is known to be hard, they evaluate the proposedmethod on synthetic datasets, or by measuring the standarddeviation of the difference in hand/object poses during agrasping scenario.

[62] considers the problem of tracking a deformable ob-ject in interaction with a hand, by optimizing an energyfunction on the appearance and the kinematics of the hand,together with hand+object contact configurations. How-ever, it is evaluated quantitatively only on synthetic images,which points to the difficulty of evaluation on real data. Inaddition, they only consider scenarios where the hand is vis-ible from a top view, restricting the range of the hand poses

and not allowing occlusions.Very recently, [26] uses a coarse hand pose estimation to

retrieve the 3D pose and shape of hand-held objects. How-ever, they only consider a specific type of object and do notestimate the object pose. [19] presents a model with con-tact loss that considers physically feasible hand+object in-teraction to improve grasp quality. However, to estimate3D hand pose, they predict PCA components for the pose,which results in lower accuracy compared to ours, as ourexperiments show. [60] proposes a deep model to jointlypredict 3D hand and object poses from egocentric views,but the absence of physical constraints might result in in-feasible grasps.

2.5. Hand+Object Datasets

Several datasets for hand+object interactions have al-ready been proposed. Many works provide egocentric RGBor RGB-D sequences for action recognition [3, 6, 7, 14, 49].However, they focus on grasp and action labels and do notprovide 3D poses. [11,33,50,62] generate synthetic datasetswith 3D hand pose annotations, but fine interaction betweena hand and an object remains difficult to generate accurately.

[63,65] captured sequences in the context of hand+handand hand+object interaction, with 2D hand annotationsonly. [34] collected a dataset of real RGB images of handsholding objects. They also provide 2D joint annotationsof pairs of non-occluded and occluded hands, by removingthe object from the grasp of the subject, while maintainingtheir hand in the same pose. [17] proposes two datasets, ahand+object segmentation dataset, and a hand+object poseestimation dataset. However, for both datasets, the back-ground pixels have been set to zero, and the training im-ages only consist of a hand interacting with a tennis ball.They provide hand pose annotations and object positions,by manually labelling the joints and using a generativemethod to refine the joint positions. [22] generate a largescale dataset with full body pose and hand pose annotationsin a multi-view setup. They use a generative approach to fitthe body and hand models to 3D keypoints and point cloud.However, their dataset focuses on total body pose annota-tion and not hand+object interactions exclusively and doesnot provide object pose annotations.

[54] proposed an RGB-D dataset of a hand manipulat-ing a cube, which contains manual ground truth for bothfingertip positions and 3D poses of the cube. [43] collecteda dataset where they measure motion and force under dif-ferent object-grasp configurations using sensors, but do notprovide 3D poses. In contrast to these previous works, [15]provides a dataset of hand and object interactions with 3Dannotations for both hand joints and object pose. They useda motion capture system made of magnetic sensors attachedto the user’s hand and to the object in order to obtain hand3D pose annotations in RGB-D video sequences. However,

DatasetNo. ofFrames

3D ObjectPose

Marker-less

RealImages Labels

No. ofObjects

No. ofSubjects

PAN [22] 675K - + + automatic - 70GAN [32] 300K - + - synthetic - -FPHA [15] 100K + (23K frames) - + automatic 26 (4 models) 6ObMan [19] 150K + + - synthetic 2.7K 20Freihand [76] 37K - + + hybrid 27 35HO-3D (ours) 78K + + + automatic 10 10

Table 1: Comparison of hand+object datasets.

this changes the appearance of the hand in the color imagesas the sensors and the tape attaching them are visible.

Very recently, [19] introduced ObMan, a large datasetof images of hands grasping objects. The images in Ob-Man dataset are synthetic and the grasps are generated us-ing an algorithm from robotics. Even more recently, [76]proposed a multi-view RGB dataset, FreiHAND, which in-cludes hand-object interactions. However, the annotationsare limited to the 3D poses and shapes of the hand. Fur-ther, [76] uses a human-in-the-loop method to obtain anno-tations from multiple RGB cameras in a green-screen back-ground environment. Our method, on the other hand is fullyautomatic, capable of working even on a single RGBD-camera setup and does not make any assumption on thebackground. The objects in our dataset are also larger thanthose in FreiHAND, thus resulting in a more challengingscenario as the occlusions are larger. The annotation accu-racy of our method is comparable to [76] as described inSection 6.1.

As illustrated in Fig. 1 and Table 1, our HO-3D datasetis the first markerless dataset providing both 3D hand jointsand 3D object pose annotations for real images, while thehand and the object are heavily occluded by each other.

3. 3D Annotation MethodWe describe below our method for annotating a sequence

T “

tpItc,Dtcqu

NCc“1

(NF

i“1of NC ˆ NF of RGB-D frames,

captured by NC cameras. The sequence captures a handinteracting with an object. Each RGB-D frame is made of acolor image Itc and a depth map Dtc.

We define the 3D hand and object poses in Section 3.1,and our general cost function in Section 3.2. We initializethe poses automatically and optimize the cost function inmultiple stages as described in Sections 4.1 and 4.2.

3.1. 3D Hand and Object Poses

We aim to estimate the 3D poses P “ tppth, ptoqu

NFt“1 for

both the hand and the object in all the images of the se-quence. We adopt the MANO hand model [51] and usethe objects from the YCB-Video dataset [70] as their corre-sponding 3D models are available and of good quality. TheMANO hand pose pth P R51 consists of 45 DoF (3 DoFfor each of the 15 finger joints) plus 6 DoF for rotation andtranslation of the wrist joint. The 15 joints together with thewrist joint form a kinematic tree with the wrist joint nodeas the first parent node. In addition to the pose parameters

pth, the hand model has shape parameters β P R10 that arefixed for a given person and we follow a method similarto [58] to estimate these parameters. More details about theshape parameter estimation are provided in the supplemen-tary material. The object pose pto P SEp3q consists of 6 DoFfor global rotation and translation.

3.2. Cost Function

We formulate the hand+object pose estimation as an en-ergy minimization problem:

P “ arg minP

NFÿ

t“1

`

EDppth, ptoq ` ECppth, p

toq˘

, (1)

where ED and EC represent the energy from data terms andconstraints, respectively. We define ED as,

EDppth, ptoq “

NCř

c“1

´

αEmaskpItc, pth, p

toq ` βEdptpDtc, p

th, p

toq`

γEj2DpItc, pthq

¯

` δE3DptDtcuc“1..NC, pth, p

toq ,

(2)where Emaskp¨q is a silhouette discrepancy term, Edptp¨q adepth residual term, Ej2Dp¨q a 2D error in hand joint loca-tions, and E3Dp¨q a 3D distance term. This last term is notabsolutely necessary, however, we observed that it signifi-cantly speeds up convergence. α, β, γ, δ are weights.

The constraints energy EC is defined as,

ECppth, ptoq “ εEjointppthq ` ζEphyppth, p

toq `

ηEtcppth, pto, p

t´1h , pt´1

o , pt´2h , pt´2

o q , (3)

where Ejointp¨q denotes a prior on the hand pose to pre-vent unnatural poses, Ephyp¨q is a physical plausibility termensuring the hand and the object do not interpenetrate,and Etcp¨q is a temporal consistency term. The terms areweighted by parameters ε, ζ and η.

We detail each of the terms in ED and EC below. Forsimplicity, we omit the frame index t from our above nota-tion except when necessary.

Silhouette discrepancy term Emask. The Emaskp¨q termcompares the silhouettes of the hand and the object mod-els rendered with the current estimated poses and their seg-mentation masks. We obtain a segmentation ScpIq of thehand and the object in the color image I of camera c usingDeepLabv3 [10] trained on images created by syntheticallyover-laying and under-laying images of hands on YCB ob-jects. More details about this step are given in the supple-mentary material. The hand and object models are renderedon the camera plane using a differentiable renderer [20],which enables computing the derivatives of Emask with re-spect to the pose parameters. The silhouette of the hand andobject rendered on camera c is denoted by RScpph, poq andthe silhouette discrepancy is defined as,

EmaskpIc, ph, poq “ ‖RScpph, poq ´ SpIcq‖2 . (4)

Depth residual term Edpt. The depth residual term issimilar to the segmentation discrepancy term:

EdptpDc, ph, poq “ Tukeyp‖RDcpph, poq ´ Dc‖q , (5)

where RDcpph, poq is the depth rendering of the hand andthe object under their current estimated poses ph and po. TheTukey function is a robust estimator that is similar to the `2loss close to 0, and constant after a threshold. It is useful tobe robust to small deviations in the scale and shape of thehand and object models and also noise in the captured depthmaps. Edpt is differentiable as we employ a differentiablerenderer [20] for rendering the depth maps.

2D Joint error term Ej2D. The 2D joint error term isdefined as,

Ej2DpIc, phq “

21ÿ

i“1

hris∥∥∥projcpJph

risq ´ Kcris∥∥∥2

, (6)

where Jphris denotes the ith 3D hand joint location under

pose ph, the projcp¨q operator projects it onto camera c,Kcris is its predicted 2D location, and hris its confidence.The 21 hand joints in Ej2Dp¨q consist of 15 finger joints, 5finger tips, and the wrist joint.

In practice, we take the Kcris as the locations of the max-imum values of heatmaps, and the hris as the maximumvalues themselves. To predict these heatmaps, we traineda CNN based on the architecture of [68]. Training datacome from an initial dataset [18] we created using a semi-automatic method. This dataset is made of 15,000 framesfrom 15 sequences in a single camera setup. We manuallyinitialized the grasp pose and object pose for the first frameof each sequence. The manipulators were asked to maintaintheir grasp poses as rigid as possible to make the registrationeasier. We then ran the optimization stages for the singlecamera case described in Section 4.2. After optimization,we augmented the resulting dataset by scaling and rotatingthe images, and adding images from the Panoptic Studiodataset [69], which contain 3D annotations for hands.

3D error term E3D. This term is not absolutely neces-sary as the depth information from all the cameras is alreadyexploited byEdpt, however it accelerates the convergence byguiding the optimization towards the minimum even fromfar away. We build a point cloud P by merging the depthmaps from the RGB-D cameras after transforming them toa common reference frame. More details on the point cloudreconstruction can be found in the supplementary material.

We segment P into an object point cloud Po and a handpoint cloud Ph using the segmentation mask ScpIq in eachcamera image. At each iteration of the optimization, foreach point Porjs of the object point cloud, we look for theclosest vertex Vorj

˚s on the object mesh, and for each point

Phrks of the hand point cloud, we look for the closest vertexVhrk

˚s on the hand mesh. E3DpP, ph, poq is then defined as,ÿ

j

∥∥Porjs ´ Vorj˚s∥∥2`ÿ

k

∥∥Phrks ´ Vhrk˚s∥∥2. (7)

Joint angle constraint Ejoint. This term imposes restric-tions on the 15 joints of the hand to ensure the resulting poseis natural. The three-dimensional rotation of a joint is pa-rameterized using the axis-angle representation in MANOmodel, resulting in 45 joint angle parameters. A commonsolution when using MANO model is to optimize in thePCA space of 3D joint angles with an `2 regularizer [4, 76]for pose coefficients. However, we observed in practice thatoptimizing Eq. 1 in PCA space had less expressibility: someof the complex grasp poses in our dataset could not be accu-rately expressed in the PCA space, which was constructedwith relatively simpler grasp poses and free-hand gestures.Instead, we optimize on joint angles directly and derive ourown limits for each of the 45 joint parameters (please referto supplementary material for these limits). As in [74], thejoint angle constraint term Ejointppthq is given by,

45ÿ

i“1

maxpai ´ aris, 0q `maxparis ´ ai, 0q , (8)

where aris denotes the ith joint angle parameter for pose ph,and ai and ai correspond to its lower and upper limits.

Physical plausibility term Ephy. During optimization,the hand model might penetrate the object model, which isphysically not possible. To avoid this, we add a repulsionterm that pushes the object and the hand apart if they in-terpenetrate each other. For each hand vertex Vhrms, theamount of penetration Γrms is taken as,

Γrms “ maxp´no`

Vorm˚s˘T `

Vhrms ´ Vorm˚s˘

, 0q ,(9)

where Vorm˚s is the vertex on object closest to hand vertex

Vhrms, and the nop¨q operator provides the normal vectorfor a vertex. In words, the amount of penetration is esti-mated by projecting the vector joining a hand vertex andits nearest object vertex onto the normal vector at the ob-ject vertex location. The physical plausibility term is thendefined as,

Ephyppth, ptoq “

ÿ

m

exp`

w Γrms˘

. (10)

We use w “ 5 in practice, and only a subsampled set ofvertices of the hand to compute Ephy efficiently.

Temporal consistency term Etc. The previous terms areall applied to each frame independently. The temporal con-sistency term Etc allows us to constrain together the poses

Grasp Pose Estimation Object Pose Estimation Multi-Frame Joint Hand-Object Pose Refinement

Bat

ch i

Init

iali

zati

on

Eq.

(11

,12)

Segmentation

RGB

Depth

Frame t1

Frame t2

Frame t3

Frame t4

Frame t Frame t+1 Frame t+2 Frame t+3

Single Camera Setup

Multi-Camera Setup

Eq. (13) Eq. (2), Eq. (1)

Camera 1 Camera 2 Camera 3 Camera 4

Single-Frame Hand-Object Pose Estimation

Frame t

Frame t+1

Frame t+2

Frame t+3

Multi-Frame Joint Hand-Object Pose Refinement

Bat

ch i

Eq. (1)Eq. (2,3)

Frame t+1

Frame t+2Frame t+3

Frame t

Figure 2: The different stages of the multi-camera and single camera setups. See Section 4 for more details.

for all the frames. We apply a 0-th and 1-st order motionmodel on both the hand and object poses:

Etcppth, pto, p

t´1h , pt´1

o , pt´2h , pt´2

o q “

‖∆th‖2 ` ‖∆t

o‖2 ` ‖∆th ´∆t´1

h ‖2 ` ‖∆to ´∆t´1

o ‖2 ,

where ∆th “ pth´pt´1

h and ∆to “ pto´pt´1

o . Since we opti-mize a sum of these terms over the sequence, this effectivelyconstrains all the poses together.

4. OptimizationOptimizing Eq. (1) is a challenging task, as it is a highly

non-convex problem with many parameters to estimate. Wetherefore perform the optimization in multiple stages asshown in Fig. 2. These stages are different for multi-cameraand single camera scenarios, and we detail them below.

4.1. Multi-Camera Setup

Initialization. In the multi-camera setup, we obtain a firstestimate p0

h for the hand pose in the first frame pt “ 0q as,

p0h “ arg min

ph

NCÿ

c“1

Ej2DpI0c , phq ` νEjointpphq . (11)

We use the Dogleg optimizer [12] to perform this optimiza-tion. A first estimate p0

o for the object pose in this frameis obtained using [47] trained by synthetically over-layinghands on YCB objects as explained in Section 3.2.

Single-frame joint pose optimization. We then obtainestimates pth and pto for all the other frames (t “ 1..NF )by tracking. We minimize

`

EDppth, ptoq `ECppth, p

toq˘

w.r.t.pth and pto, using pt´1

h and pt´1o for initialization.

Multi-frame joint pose optimization. We finally per-form a full optimization of Eq. (1) w.r.t. pth and pto for

t “ 0..NF over all the frames simultaneously using es-timates pth and pto for initialization. Due to memory con-straints, we optimize Eq. (1) in batches instead of consider-ing all the frames in sequence. We use a batch size of 20frames with α “ 20, β “ 20, γ “ 5 ˆ 10´5, δ “ 50,ε “ 100, ζ “ 50, and η “ 100, and the Adam optimizerwith learning rate of 0.01 for 100 iterations.

4.2. Single-Camera Setup

Initialization. In the single camera setup, as we assumethat the grasp pose varies marginally across the sequence,we initially make the assumption that it remains constantthroughout the sequence. In order to account for the minorchanges which occur in practice, we relax this assumptionin the latter stages of the optimization. We thus obtain initialestimates pth for the hand poses as,

tpthut “ arg mintpthut

ÿ

t

Ej2DpIt, pthq ` νEjointppthq , (12)

where the joint angle parameters are constrained to be thesame over all the frames, and only the rotation and transla-tion parameters for the wrist joint can be different. In prac-tice, we perform this optimization only over a random sub-set Ω of the frames to save time. We set ν “ 50, size of Ω to20 and use the Dogleg optimizer [12]. First estimates pto forthe object poses in Ω are obtained as for the multi-camerasetup.

From the pth and the pto, we can compute the grasp posein the object coordinate system pth:o, which is assumed to beconstant at this stage. The initial estimate of the constantgrasp pose ph:o is taken as the average of tpth:outPΩ.

Grasp pose estimation. We obtain a better estimate of thegrasp pose ph:o and object poses pto under fixed grasp pose

assumption as,

ph:o, tptoutPΩ “ arg minph:o,tptoutPΩ

ř

tPΩEDpfpto pph:oq, ptoq`

ζEphypfpto pph:oq, ptoq ` εEjointpph:oq,

(13)wrt ph:o and pto over the frames in Ω, using ph:o and pto forinitialization. fpto p¨q converts the grasp pose to the handpose in the world coordinate system given object pose pto.This optimization accounts for the mutual occlusions be-tween hand and object.

Object pose estimation. Having a good estimate ph:o forthe grasp pose, we obtain object pose estimates pto for all theframes by minimizing ED

`

fpto pph:oq, pto

˘

wrt pto over eachframe independently. We use pt´1

o to initialize the optimiza-tion over pto at frame t, except for p0

o where p0o is used. Note

that the hand pose is not optimized in this stage.

Multi-frame joint hand+object pose refinement. In thisfinal stage, we allow variations in the grasp pose acrossframes and introduce temporal constraints. We thus opti-mize Eq. (1) w.r.t. tppth, p

toqu

NFt“1 over all the frames simulta-

neously, using pose parameters pto, pth “ fpto pph:oq estimatedin the previous stages as initialization for pto and pth.

5. Monocular RGB based 3D Hand PoseFor establishing a baseline on our proposed dataset for

single RGB image based hand pose prediction, we usea CNN architecture based on a Convolutional Pose Ma-chine (CPM) [68] to predict the 2D hand joint locationstkiui“1..21. In addition, we also predict the root-relativehand joint directions tdiui“1..20, by adding an additionalstage at the end of the CPM and replacing the last layer witha fully connected layer. More details on the architecture areprovided in the supplementary material. The 3D joint lo-cations and shape parameters of the hand are then obtainedby fitting a MANO model to these predictions. The lossfunction for this fitting procedure is:

21ÿ

i“1

ki ´ ki2 ` ρ20ÿ

i“1

`

1´ di ¨ di˘

` σEjointpphq ` τβ2 ,

(14)

where di “phri`1s´phr1sphri`1s´phr1s

, ki “ proj`

Jphris˘

and Ejoint isdefined in Eq. (8). We use ρ “ 10, σ “ 5, and τ “ 1.

6. Benchmarking HO-3DIn this section, we evaluate both our annotation method

and our baseline for hand pose prediction from a singlecolor image in hand+object interaction scenarios. We usedour 3D pose annotation method to annotate 68 sequences,totalling 77,558 frames of 10 different users manipulating

one among 10 different objects from the YCB dataset. Theimage sizes are 640ˆ480 pixels for both the color and depthcameras, and we used 5 synchronized cameras in our multi-camera setup. The cameras were synchronized with an ac-curacy of 5ms.

6.1. Evaluation of the Annotation Method

For validating the accuracy of our annotation method,we manually annotated the 3D locations of the 3D joints inrandomly selected frames of a sequence, by relying on theconsolidated point cloud from the 5 cameras. We then com-pared these locations to the ones predicted with our method(explained in Section 4.1) using the multi-camera setup.

As shown in the last column of Table 2, our methodachieves an average joint error accuracy of lower than 8mmon average, with an Area Under the Curve metric (AUC) of0.79. This metric is comparable with the results reported forthe recent FreiHAND dataset [76] (AUC=0.791). Note thatthe occlusions in our dataset are higher due to larger objectsand that we do not use green screens.

To analyze the influence of the different terms in Eq. (1),we run the optimization of Eq. (1) by enabling only a sub-set of these terms, and report the results in Table 2. WhileEsilh and Edpt terms alone cannot provide good pose esti-mates, together they provide better estimates as it leads toa loss function with less local minima. The E3D term pro-vides a minor improvement in estimates but speeds up theconvergence. Though the physical plausibility term Ephydoes not help in improving the pose estimates, it results inmore natural grasps. The last two columns show the effectof multi-frame based joint optimization compared to single-frame based optimization when all the terms are considered.The multi-frame multi-camera based optimization over allthe terms improves the accuracy by about 15%.

The accuracy of the single camera based annotationmethod is calculated by considering the annotations fromthe multi-camera method as ground truth for a given se-quence. More specifically, for a sequence of 1000 frames,we compute the average difference between hand+objectmesh vertices obtained from single and multi-camera se-tups. Further, we calculate the accuracy after each stage ofthe single-camera setup. The results are given in Table 3.The estimated poses with these two methods are consistentwith each other with an average mesh error of 0.77cm and0.45cm for hand and object, respectively. The final refine-ment stage yields a 15% improvement in accuracy.

6.2. Evaluation of Hand Pose Prediction Method

We trained our single frame hand pose prediction methodexplained in Section 5 on 66,034 frames from our HO-3Ddataset. We evaluated it on a test set of 13 sequences cap-tured from different viewpoints and totaling 11,524 frames.The test set sequences also contain subjects and objects not

Terms Initialization Single-frame Optimization Multi-frameOpt. (Eq. 1)Esilh Edpt Esilh ` Edpt Esilh ` Edpt ` E3D Esilh ` Edpt ` E3D ` Ephy Esilh ` Edpt ` E3D ` Ephy ` Etc

mean (std) 4.20 (˘3.32) 1.17 (˘1.12) 2.22 (˘1.22) 1.04 (˘0.43) 0.98 (˘0.40) 0.99 (˘0.40) 0.92 (˘0.34) 0.77 (˘0.29)

Table 2: Evaluation of the accuracy for the multi-camera setup. We report the average hand-joint errors (in cm) for differentcombinations of the terms in Eq. (1). The final error is comparable to the recent FreiHAND dataset [76].

Stages Init. Grasp Pose Est. Object Pose Est. Refinement

Hand 5.40 3.60 0.91 0.77Object 4.02 4.02 0.52 0.45

Table 3: Evaluation of the accuracy for the single-camerasetup. The accuracy (average mesh error in cm) is measuredat each stage of optimization by comparing with the anno-tations from multi-camera setup. The results show that theannotation quality of our single camera method is similar tothat of the multi-camera setup.

Method Mesh ErrorÓ F@5mmÒ F@15mmÒ Joint ErrorÓ

Joints2D 1.14 0.49 0.93 3.14Joints2D + Dir. Vec. 1.06 0.51 0.94 3.04[19] 1.10 0.46 0.93 3.18

Table 4: Evaluation of different methods for single framehand pose prediction. The Mesh Error (in cm) and F-score are obtained after aligning the predicted meshes withground truth meshes. The Mean joint error (in cm) is ob-tained after aligning the position of the root joint and over-all scale with the ground truth. Hand pose prediction usingjoint direction predictions along with 2D joint predictionsprovides better accuracy than directly predicting the MANOparameters as in [19].

present in the training set.We report three different metrics from previous works:

Mean joint position error after aligning the position of theroot joint and global scale with ground truth [75]; Mesherror measuring the average Euclidean distance betweenpredicted and ground truth mesh vertices [76]; and the F -score [76], defined as the harmonic mean between recalland precision between two meshes given a distance thresh-old. The mesh error and F-score are obtained after aligningthe predicted meshes using Procrustes alignment with theground truth meshes and hence does not measure the ac-curacy of wrist joint rotation. The mean joint error on theother hand considers wrist joint location as the 3D pointsare not rotated before evaluation.

To understand the effect of joint direction predictions onthe overall accuracy, we evaluate the results of the MANOfitting by dropping the second term in Eq. (14). We alsocompare our results with the hand branch of [19], a veryrecent work that predicts the MANO pose and shape pa-rameters directly from a single RGB image, retrained onour dataset. As shown in Table 4, predicting joint direc-tions along with 2D joint locations significantly improvesthe hand pose estimation accuracy. It can also be inferredthat predicting 2D hand joint locations and fitting MANO

Figure 3: Qualitative results of our single color image handpose estimation method. It can recover hand poses evenwhen the hand is heavily occluded by objects and in clut-tered scenes. The last row shows it can handle unseen ob-jects.

model to them is more accurate than direct MANO param-eter predictions as in [19]. Qualitative results are shown inFig. 3. The last row shows that our method robustly predictshand poses even when interacting with unknown objects.

7. ConclusionWe introduced a fully automatic method to annotate im-

ages of a hand manipulating an object with their 3D poses,even under large occlusions, by exploiting temporal con-sistency. We also introduced the first markerless dataset ofcolor images for benchmarking 3D hand+object pose esti-mation. To demonstrate the usefulness of our dataset, weproposed a method for predicting the 3D pose of the handfrom a single color image. Another future application is thejoint estimation of hand+object poses from a single RGBframe.

The lack of high quality segmentations (we had to usea synthetic dataset as explained in Section 3) of the handsometimes affects accuracy. Improving these segmentationsand/or introducing a attraction term and physics constraintsas in [54, 63] would further improve our annotations.

8. AcknowledgementThis work was supported by the Christian Doppler Lab-

oratory for Semantic 3D Computer Vision, funded in partby Qualcomm Inc.

HOnnotate: A method for 3D Annotation of Hand and Object PosesSupplementary Material

Shreyas Hampali1, Mahdi Rad1, Markus Oberweger1, and Vincent Lepetit2,1

1Institute for Computer Graphics and Vision, Graz University of Technology, Austria2LIGM, Ecole des Ponts, Univ Gustave Eiffel, CNRS, Marne-la-Valle, France

hampali, rad, oberweger, lepetit@icg.tugraz.at, Project Page: https://www.tugraz.at/index.php?id=40231

S.1. Hand Pose Estimation from Single ColorImage

Fig. 4 shows the architecture of our hand pose estimatorfrom a single frame. Given an image of the hand centeredin the image window, we first extract features using the con-volutional layers of VGG [53], and then similar to [9] usinga multi-stage CNN, we predict heatmaps for the 2D handjoint locations and finally joint direction vectors with re-spect to wrist joint. The hand detection can be done usingsegmentation as described in Section S.6.

S.2. Hand Pose Estimation for Hand Interac-tion with Unseen Objects

Knowing the objects in advance can help to improve theperformances of the estimated 3D hand pose while handinteracts with objects, however, in practice, the hand canmanipulate any arbitrary objects. We have tested our handpose estimator trained on our annotations, and tested on se-quences where a hand is manipulating objects not present inthe annotated images. As shown in Fig. 8, our pose estima-tor performs well on these sequences.

S.3. Hand Shape EstimationThe MANO hand shape parameters β P R10 were esti-

mated for each human manipulator in our HO-3D dataset.The shape parameters are estimated from a sequence Φ ofhand only poses using a method similar to [58] in two steps.More exactly, the pose of hand pth in the sequence is firstestimated for each frame t using a mean pose βmean aspth “ arg minph EHpph, βmeanq, where,

EHpph, βmeanq “EDpph, βmeanq ` εEjointpphq` (15)

ηEtcpph, pt´1h , pt´2

h q.

EDpph, βmeanq represents the data term defined in Eq. 2 ofthe paper where hand is rendered with pose parameters ph

Joint Index Middle Pinky Ring Thumb

MCP(0.00, 0.45)(-0.15, 0.20)(0.10, 1.80)

(0.00, 0.00)(-0.15, 0.15)(0.10, 2.00)

(-1.50, -0.20)(-0.15, 0.60)(-0.10, 1.60)

(-0.50, -0.40)(-0.25, 0.10)(0.10, 1.80)

(0.00, 2.00)(-0.83, 0.66)(0.00, 0.50)

PIP(-0.30, 0.20)(0.00, 0.00)(0.00, 0.20)

(-0.50, -0.20)(0.00, 0.00)(0.00, 2.00)

(0.00, 0.00)(-0.50, 0.60)(0.00, 2.00)

(-0.40, -0.20)(0.00, 0.00)(0.00, 2.00)

(-0.15, 1.60)(0.00, 0.00)(0.00, 0.50)

DIP(0.00, 0.00)(0.00, 0.00)(0.00, 1.25)

(0.00, 0.00)(0.00, 0.00)(0.00, 1.25)

(0.00, 0.00)(0.00, 0.00)(0.00, 1.25)

(0.00, 0.00)(0.00, 0.00)(0.00, 1.25)

(0.00, 0.00)(-0.50, 0.00)(-1.57, 1.08)

Table 5: Empirically derived minimum and maximum val-ues for the joint angle parameters used in our implementa-tion.

and shape parameters βmean. Ejoint and Etc are explainedin Section 3.2 of the paper. At each frame, the pose param-eters are initialized with pt´1

h . The personalized hand shapeparameters are then obtained as,

β˚ “ arg minβÿ

tPΦ

minpth

EHppth, βq, (16)

where the pose parameters are initialized with the valuesobtain in the first step (pth).

S.4. Joint Angle ConstraintsThe maximum and minimum limits on the joint angle pa-

rameters used in Eq. (8) of the paper are provided in Table 5.

S.5. Point Cloud from Multiple CamerasThe E3D term in Section 3.2 of the paper uses the com-

bined point cloud P from all the RGB-D cameras. LetPc denote the point cloud corresponding to camera c andMc1,c2 denote the relative pose between two cameras c1 andc2. The consolidated point cloud P is then obtained as,

P “ rP0, Mc1,c0 ¨P1, Mc2,c0 ¨P2, ..., McN ,c0 ¨PN s , (17)

where r¨, ¨s represents concatenation of point clouds.

9

Input Image

VGG[C1_1:C4_4]

Stage 2

7×7C

128

7×7C

128

7×7C

128

7×7C

128

7×7C

128

7×7C

128

1×1C1

32×32Stage 1 Stage 3 FC1256

FC260

Joints20×3

256×256L2 Loss

32×3232×32

Down-sampling

7×7C

128P

2×2

5×5C

256P

2×2

1×1C

512AP8×8

Figure 4: Architecture of our hand pose estimator from a single color image. Given an input image of hand centered in theimage, we extract the features using the convolutional layers of VGG [53] (Conv1 1 to Conv4 4). Similarly to [9], we thenpredict heatmaps for the joint locations in multi-stages. The architecture for the different stages are all the same. C denotesa convolutional layer with the number of filters and the filter size inscribed; FC, a fully-connected layer with the number ofneurons; P and AP denote max-pooling and average pooling with their sizes, respectively.

Figure 5: Synthetic training images used for training thehand-object segmentation network.

Figure 6: Example of hand and object segmentation ob-tained with DeepLabV3. Left: input image; Right: hand(green) and object (purple) segmentation.

S.6. Hand-Object Segmentation Network

The segmentation maps for the hand and object are ob-tained from a DeepLabV3 [10] network trained on syntheticimages of hand and objects. The synthetic images are ob-tained by over-laying and under-laying images of hands onimages of objects at random locations and scales. We usethe object masks provided by [1]. The segmented handswere obtained using an RGB-D camera by applying simpledepth thresholding. We also use additional synthetic handimages from the RHD dataset [75]. A few example imagesfrom the training data are shown in Fig. 5. We use 100Ktraining images with augmentations. Fig. 6 shows segmen-tation of hand and object using the trained DeepLabV3 net-work.

S.7. Automatic Initialization

Figure 7: Accuracy of keypoint prediction, describedin Section 3.2 of the paper when trained with PAN [22]dataset alone and PAN + our annotations. The accuracyis measured in percentage of correct 2D keypoints given athreshold. Only 15,000 images from our HO-3D dataset areused in training. Due to the presence of object occlusions,a network trained on hands-only dataset is less accurate inpredicting keypoints when compared with a network trainedwith hand+object data.

As explained in Section 4.1 of the paper, a keypoint pre-diction network based on convolutional pose machine [68]is used to obtain initialization for hand poses. Such anetwork is trained with our initial hand+object datasetof 15,000 images together with images from hand-onlyPAN [22] dataset. Fig. 7 compares the accuracy of networkin predicting keypoints in hand-object interaction scenar-ios when trained with hands-only dataset and hands+objectdataset. Our initial HO-3D dataset helps in obtaining a moreaccurate network for predicting keypoints and hence resultsin better initialization.

S.8. Dataset Details

We annotated 77,558 frames of 68 sequences hand-object interaction of 10 persons with different hand shape.On average there are 1200 frames per sequences. 16 se-quences are captured and annotated in a single camera, and52 sequences for the multi-camera setup.

Hand+Object. The participants are asked to perform ac-tions with objects. The grasp poses vary between framesin a sequence in the multi-camera setup and remain almostrigid in the single camera setup.

Participants. The participants are between 20 and 40years old, 7 of them are males and 3 are females. In total,10 hand shapes are considered for the annotations.

Objects. We aimed to choose 10 different objects fromthe YCB dataset [70] that are used in daily life. As shownin Fig. 11, we have a wide variety of sizes such as largeobjects (e.g. Bleach) that cause large hand occlusion, orthe objects that make grasping and manipulation difficult(e.g. Scissors), while these are not the case in the existinghand+object datasets.

Multi-Camera Setup. We use 5 calibrated RGB-D cam-eras, in our multi-camera setup. The cameras are locatedat different angles and locations. Our cameras are syn-chronized with a precision of about 5 ms. The scenes arecluttered with objects, and the backgrounds vary betweenscenes.

Figs. 9 and 10 show some examples of the 3D annotatedframes for both hand and object from our proposed dataset,HO-3D.

References[1] YCB Benchmarks Object and Model Set.

http://ycbbenchmarks.org/. 2, 10[2] Luca Ballan, Aparna Taneja, Jrgen Gall, Luc Van Gool, and

Marc Pollefeys. Motion capture of hands in action usingdiscriminative salient points. In European Conference onComputer Vision (ECCV), pages 640–653, Firenze, October2012. 3

[3] Sven Bambach, Stefan Lee, David J. Crandall, and Chen Yu.Lending a Hand: Detecting Hands and Recognizing Activ-ities in Complex Egocentric Interactions. In The IEEE In-ternational Conference on Computer Vision (ICCV), pages1949–1957, 2015. 3

[4] Adnane Boukhayma, Rodrigo de Bem, and Philip HS Torr.3d hand shape and pose from images in the wild. In TheIEEE Conference on Computer Vision and Pattern Recogni-tion (CVPR), pages 10843–10852, 2019. 5

[5] Anders Glent Buch, Lilita Kiforenko, and Dirk Kraft. Rota-tional subgroup voting and pose clustering for robust 3d ob-ject recognition. In The IEEE International Conference OnComputer Vision (ICCV), pages 4137–4145, United States,2017. 2

[6] Ian M. Bullock, Thomas Feix, and Aaron M. Dollar. TheYale Human Grasping Dataset: Grasp, Object, and Task Datain Household and Machine Shop Environments. The Interna-tional Journal of Robotics Research, 34(3):251–255, 2015. 3

[7] Minjie Cai, Kris M. Kitani, and Yoichi Sato. A ScalableApproach for Understanding the Visual Structures of HandGrasps. In The IEEE International Conference on Roboticsand Automation (ICRA), 2015. 3

[8] Qixin Cao and Haoruo Zhang. Combined Holistic and Lo-cal Patches for Recovering 6D Object Pose. In The IEEEInternational Conference on Computer Vision (ICCV), pages2219–2227, 2017. 2

[9] Z. Cao, T. Simon, S.-E. Wei, and Y. Sheikh. Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields. 2017.9, 10

[10] Liang-Chieh Chen, George Papandreou, Florian Schroff, andHartwig Adam. Rethinking Atrous Convolution for Seman-tic Image Segmentation. CoRR, abs/1706.05587, 2017. 4,10

[11] Chiho Choi, Sang Ho Yoon, Chin-Ning Chen, and KarthikRamani. Robust Hand Pose Estimation During the Interac-tion with an Unknown Object. In The IEEE InternationalConference on Computer Vision (ICCV), pages 3142–3151,2017. 3

[12] Andrew R. Conn, Nicholas I. M. Gould, and Philippe L.Toint. Trust-Region Methods. SIAM, 2000. 6

[13] Martin de La Gorce, David J. Fleet, and Nikos Paragios.Model-Based 3D Hand Pose Estimation from MonocularVideo. IEEE Trans. Pattern Anal. Mach. Intell.(PAMI),33(9):1793–1805, 2011. 2

[14] Alireza Fathi, Xiaofeng Ren, and James M. Rehg. Learn-ing to Recognize Objects in Egocentric Activities. In TheIEEE Conference on Computer Vision and Pattern Recogni-tion (CVPR), pages 3281–3288, 2011. 3

[15] Guillermo Garcia-Hernando, Shanxin Yuan, SeungryulBaek, and Tae-Kyun Kim. First-Person Hand Action Bench-mark with RGB-D Videos and 3D Hand Pose Annotations.In The IEEE Conference on Computer Vision and PatternRecognition (CVPR), pages 409–419, 2018. 1, 2, 3, 4

[16] Liuhao Ge, Yujun Cai, Junwu Weng, and Junsong Yuan.Hand PointNet: 3D Hand Pose Estimation Using Point Sets.In The IEEE Conference on Computer Vision and PatternRecognition (CVPR), pages 8417–8426, 2018. 2

[17] Duncan Goudie and Aphrodite Galata. 3D Hand-ObjectPose Estimation from Depth with Convolutional Neural Net-works. In IEEE International Conference on Automatic Face& Gesture Recognition, 2017. 3

[18] Shreyas Hampali, Markus Oberweger, Mahdi Rad, andVincent Lepetit. HO-3D: a Multi-User, Multi-ObjectDataset for Joint 3D Hand-Object Pose Estimation. CoRR,abs/1907.01481, 2019. 5

[19] Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kale-vatykh, Michael J. Black, Ivan Laptev, and Cordelia Schmid.

Figure 8: Qualitative results of 3D hand pose estimation of hand manipulating unseen objects. Our pose estimator trained onthe HO-3D dataset is still able to predict accurate 3D poses when interacting with new objects.

Figure 9: Some examples of the 3D annotated frames for both hand and object from our proposed dataset, HO-3D.

../

Figure 10: Some examples of the 3D annotated frames for both hand and object from our proposed dataset, HO-3D.

Banana Bleach Cracker Box Driller Mug

Mustard Bottle Pitcher Potted Meat Scissors Sugar Box

Figure 11: 10 objects of the YCB dataset [70] that we use for our dataset HO-3D.

Learning Joint Reconstruction of Hands and ManipulatedObjects. In The IEEE Conference on Computer Vision andPattern Recognition (CVPR), pages 11807–11816, 2019. 1,2, 3, 4, 8

[20] Paul Henderson and Vittorio Ferrari. Learning Single-Image3D Reconstruction by Generative Modelling of Shape, Poseand Shading. International Journal of Computer Vision(IJCV), pages 1573–1405, 2019. 4, 5

[21] Yinlin Hu, Joachim Hugonot, Pascal Fua, and Mathieu Salz-mann. Segmentation-Driven 6d Object Pose Estimation.In The IEEE Conference on Computer Vision and PatternRecognition (CVPR), pages 3385–3394, 2019. 2

[22] Hanbyul Joo, Tomas Simon, and Yaser Sheikh. Total Cap-ture: A 3D Deformation Model for Tracking Faces, Hands,and Bodies. In The IEEE Conference on Computer Visionand Pattern Recognition (CVPR), pages 8320–8329, 2018.3, 4, 10

[23] Wadim Kehl, Fausto Milletari, Federico Tombari, SlobodanIlic, and Nassir Navab. Deep Learning of Local RGB-DPatches for 3D Object Detection and 6D Pose Estimation.In European Conference on Computer Vision (ECCV), pages205–220, 2016. 2

[24] Cem Keskin, Furkan Kirac, Yunus Emre Kara, and LaleAkarun. Hand Pose Estimation and Hand Shape Classifi-cation Using Multi-Layered Randomized Decision Forests.In European Conference on Computer Vision (ECCV), pages852–863, 2012. 2

[25] Diederik P. Kingma and Jimmy Ba. Adam: A Method forStochastic Optimization. In The International Conferenceon Learning Representations (ICLR), 2015. 2

[26] Mia Kokic, Danica Kragic, and Jeannette Bohg. Learning toEstimate Pose and Shape of Hand-Held Objects from RGBImages. In The IEEE/RSJ International Conference on In-telligent Robots and Systems (IROS), pages 3980–3987, 112019. 3

[27] Nikolaos Kyriazis and Antonis A. Argyros. Physically Plau-sible 3D Scene Tracking: The Single Actor Hypothesis.In The IEEE Conference on Computer Vision and PatternRecognition (CVPR), 2013. 1, 3

[28] Nikolaos Kyriazis and Antonis A. Argyros. Scalable 3DTracking of Multiple Interacting Objects. In The IEEEConference on Computer Vision and Pattern Recognition(CVPR), 2014. 3

[29] Stan Melax, Leonid Keselman, and Sterling Orsten. Dynam-ics Based 3D Skeletal Hand Tracking. In Proceedings ofGraphics Interface, 2013. 2

[30] Andrew T. Miller and Peter K. Allen. Graspit! a VersatileSimulator for Robotic Grasping. Robotics Automation Mag-azine, pages 110–122, 2004. 1

[31] Chaitanya Mitash, Abdeslam Boularias, and Kostas E.Bekris. Improving 6D Pose Estimation of Objects in Clut-ter via Physics-Aware Monte Carlo Tree Search. In TheIEEE International Conference on Robotics and Automation(ICRA), pages 1–8, 2018. 2

[32] Franziska Mueller, Florian Bernard, Oleksandr Sotny-chenko, Dushyant Mehta, Srinath Sridhar, Dan Casas, andChristian Theobalt. Ganerated Hands for Real-Time 3DHand Tracking from Monocular RGB. In The IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR),pages 49–59, 2018. 1, 2, 3, 4

[33] Franziska Mueller, Dushyant Mehta, Oleksandr Sotny-chenko, Srinath Sridhar, Dan Casas, and Christian Theobalt.Real-Time Hand Tracking Under Occlusion from an Egocen-tric Rgb-D Sensor. In The IEEE International Conference onComputer Vision (ICCV), 2017. 1, 2, 3

[34] Battushig Myanganbayar, Cristina Mata, Gil Dekel, BorisKatz, Guy Ben-Yosef, and Andrei Barbu. Partially OccludedHands: A Challenging New Dataset for Single-Image HandPose Estimation. In The Asian Conference on Computer Vi-sion (ACCV), pages 85–98, 2018. 3

[35] Natalia Neverova, Christian Wolf, Florian Nebout, and Gra-ham W. Taylor. Hand Pose Estimation through Semi-Supervised and Weakly-Supervised Learning. Comput. Vis.Image Underst. (CVIU), 164:56–67, 2017. 2

[36] Markus Oberweger and Vincent Lepetit. DeepPrior++: Im-proving Fast and Accurate 3D Hand Pose Estimation. In TheIEEE International Conference on Computer Vision Work-shops (ICCV), pages 585–594, 2017. 1, 2

[37] Markus Oberweger, Mahdi Rad, and Vincent Lepetit. Mak-ing Deep Heatmaps Robust to Partial Occlusions for 3D Ob-ject Pose Estimation. In European Conference on ComputerVision (ECCV), pages 125–141, 2018. 2

[38] Markus Oberweger, Gernot Riegler, Paul Wohlhart, and Vin-cent Lepetit. Efficiently Creating 3D Training Data for FineHand Pose Estimation. In The IEEE Conference on Com-puter Vision and Pattern Recognition (CVPR), pages 4957–4965, 2016. 1

[39] Iasonas Oikonomidis, Nikolaos Kyriazis, and Antonis A. Ar-gyros. Full DoF Tracking of a Hand Interacting with anObject by Modeling Occlusions and Physical Constraints.In The IEEE International Conference on Computer Vision(ICCV), pages 2088–2095, 2011. 3

[40] Paschalis Panteleris, Nikolaos Kyriazis, and Antonis A. Ar-gyros. 3D Tracking of Human Hands in Interaction with Un-known Objects. In Proceedings of the British Machine VisionConference 2015 (BMVC), pages 123.1–123.12, 2015. 3

[41] Paschalis Panteleris, Iason Oikonomidis, and Antonis A. Ar-gyros. Using a Single RGB Frame for Real Time 3D HandPose Estimation in the Wild. In The IEEE Winter Conferenceon Applications of Computer Vision (WACV), pages 436–445, 2018. 2

[42] Sida Peng, Yuan Liu, Qixing Huang, Xiaowei Zhou, and Hu-jun Bao. Pvnet: Pixel-Wise Voting Network for 6DoF PoseEstimation. In The IEEE Conference on Computer Visionand Pattern Recognition (CVPR), pages 4561–4570, 2019. 2

[43] Tu-Hoa Pham, Nikolaos Kyriazis, Antonis A. Argyros, andAbderrahmane Kheddar. Hand-Object Contact Force Esti-mation from Markerless Visual Tracking. IEEE Trans. Pat-tern Anal. Mach. Intell.(PAMI), 40(12):2883–2896, 2018. 3

[44] Tu-Hoa Pham, Abderrahmane Kheddar, Ammar Qammaz,and Antonis A. Argyros. Capturing and Reproducing Hand-Object Interactions through Vision-Based Force Sensing. InObject Understanding for Interaction, 2015. 3

[45] Tu-Hoa Pham, Abderrahmane Kheddar, Ammar Qammaz,and Antonis A. Argyros. Towards Force Sensing from Vi-sion: Observing Hand-Object Interactions to Infer Manipu-lation Forces. In The IEEE Conference on Computer Visionand Pattern Recognition (CVPR), 2015. 3

[46] Chen Qian, Xiao Sun, Yichen Wei, Xiaoou Tang, and JianSun. Realtime and Robust Hand Tracking from Depth.In The IEEE Conference on Computer Vision and PatternRecognition (CVPR), 2014. 1, 2

[47] Mahdi Rad and Vincent Lepetit. BB8: A Scalable, Accurate,Robust to Partial Occlusion Method for Predicting the 3DPoses of Challenging Objects Without Using Depth. In TheIEEE International Conference on Computer Vision (ICCV),2017. 1, 6

[48] Mahdi Rad, Markus Oberweger, and Vincent Lepetit. Do-main Transfer for 3D Pose Estimation from Color ImagesWithout Manual Annotations. In The Asian Conference onComputer Vision (ACCV), 2018. 1, 3

[49] Gregory Rogez, James Steven Supancic III, and Deva Ra-manan. Understanding Everyday Hands in Action fromRGB-D Images. In The IEEE International Conference onComputer Vision (ICCV), 2015. 3

[50] Gregory Rogez, Maryam Khademi, James Steven SupancicIII, J. M. M. Montiel, and Deva Ramanan. 3D Hand PoseDetection in Egocentric RGB-D Images. In European Con-ference on Computer Vision (ECCV), 2014. 3

[51] Javier Romero, Dimitrios Tzionas, and Michael J. Black.Embodied Hands: Modeling and Capturing Hands and Bod-ies Together. ACM Trans. Graph., 36(6):245, 2017. 2, 4

[52] Toby Sharp, Cem Keskin, Duncan P. Robertson, JonathanTaylor, Jamie Shotton, David Kim, Christoph Rhemann, IdoLeichter, Alon Vinnikov, Yichen Wei, Daniel Freedman,Pushmeet Kohli, Eyal Krupka, Andrew W. Fitzgibbon, andShahram Izadi. Accurate, Robust, and Flexible Real-TimeHand Tracking. In Proceedings of the 33rd Annual ACMConference on Human Factors in Computing Systems, CHI,pages 3633–3642, 2015. 2

[53] K. Simonyan and A. Zisserman. Very Deep Convolu-tional Networks for Large-Scale Image Recognition. CoRR,abs/1409.1556, 2014. 9, 10

[54] Srinath Sridhar, Franziska Mueller, Michael Zollhofer, DanCasas, Antti Oulasvirta, and Christian Theobalt. Real-TimeJoint Tracking of a Hand Manipulating an Object from RGB-D Input. In European Conference on Computer Vision(ECCV), volume 9906, pages 294–310, 2016. 1, 2, 3, 8

[55] Srinath Sridhar, Antti Oulasvirta, and Christian Theobalt. In-teractive Markerless Articulated Hand Motion Tracking Us-ing RGB and Depth Data. In The IEEE International Confer-ence on Computer Vision (ICCV), pages 2456–2463, 2013. 2

[56] Xiao Sun, Yichen Wei, Shuang Liang, Xiaoou Tang, and JianSun. Cascaded Hand Pose Regression. In The IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR),pages 824–832, 2015. 1

[57] Martin Sundermeyer, Zoltan-Csaba Marton, MaximilianDurner, Manuel Brucker, and Rudolph Triebel. Implicit 3DOrientation Learning for 6D Object Detection from RGB Im-ages. In European Conference on Computer Vision (ECCV),pages 712–729, 2018. 1

[58] David Joseph Tan, Thomas J. Cashman, Jonathan Tay-lor, Andrew W. Fitzgibbon, Daniel Tarlow, Sameh Khamis,Shahram Izadi, and Jamie Shotton. Fits Like a Glove: Rapidand Reliable Hand Shape Personalization. In The IEEEConference on Computer Vision and Pattern Recognition(CVPR), pages 5610–5619, June 2016. 4, 9

[59] Danhang Tang, Hyung Jin Chang, Alykhan Tejani, and Tae-Kyun Kim. Latent Regression Forest: Structured Estimationof 3D Articulated Hand Posture. In The IEEE Conferenceon Computer Vision and Pattern Recognition (CVPR), pages3786–3793, 2014. 1

[60] Bugra Tekin, Federica Bogo, and Marc Pollefeys. H+O: Uni-fied Egocentric Recognition of 3D Hand-Object Poses and

Interactions. In The IEEE Conference on Computer Visionand Pattern Recognition (CVPR), pages 4511–4520, 2019. 3

[61] Jonathan Tompson, Murphy Stein, Yann LeCun, and KenPerlin. Real-Time Continuous Pose Recovery of HumanHands Using Convolutional Networks. ACM Trans. Graph.,33:169:1–169:10, 2014. 2

[62] Aggeliki Tsoli and Antonis A. Argyros. Joint 3D Trackingof a Deformable Object in Interaction with a Hand. In Eu-ropean Conference on Computer Vision (ECCV), pages 504–520, 2018. 3

[63] Dimitrios Tzionas, Luca Ballan, Abhilash Srikantha, PabloAponte, Marc Pollefeys, and Juergen Gall. CapturingHands in Action Using Discriminative Salient Points andPhysics Simulation. International Journal of Computer Vi-sion (IJCV), 118(2):172–193, 2016. 1, 3, 8

[64] Dimitrios Tzionas and Juergen Gall. 3D Object Reconstruc-tion from Hand-Object Interactions. In The IEEE Interna-tional Conference on Computer Vision (ICCV), pages 729–737, 2015. 3

[65] Dimitrios Tzionas, Abhilash Srikantha, Pablo Aponte, andJuergen Gall. Capturing Hand Motion with an RGB-D Sen-sor, Fusing a Generative Model with Salient Points. InGerman Conference on Pattern Recognition (GCPR), pages277–289, 2014. 2, 3

[66] Chengde Wan, Thomas Probst, Luc Van Gool, and AngelaYao. Self-Supervised 3D Hand Pose Estimation throughTraining by Fitting. In The IEEE Conference on Com-puter Vision and Pattern Recognition (CVPR), pages 10853–10862, June 2019. 3

[67] Robert Y. Wang, Sylvain Paris, and Jovan Popovic. 6DHands: Markerless Hand-Tracking for Computer Aided De-sign. In ACM Symposium on User Interface Software andTechnology, pages 549–558, 2011. 3

[68] Shih-En Wei, Varun Ramakrishna, Takeo Kanade, and YaserSheikh. Convolutional Pose Machines. In The IEEE Confer-ence on Computer Vision and Pattern Recognition (CVPR),pages 4724–4732, 2016. 5, 7, 10

[69] Donglai Xiang, Hanbyul Joo, and Yaser Sheikh. MonocularTotal Capture: Posing Face, Body, and Hands in the Wild.In The IEEE Conference on Computer Vision and PatternRecognition (CVPR), pages 10965–10974, 2019. 5

[70] Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, andDieter Fox. PoseCNN: A Convolutional Neural Network for6D Object Pose Estimation in Cluttered Scenes. In Robotics:Science and Systems XIV (RSS), 2018. 2, 4, 11, 15

[71] Chi Xu, Lakshmi Narasimhan Govindarajan, Yu Zhang, andLi Cheng. Lie-X: Depth Image Based Articulated ObjectPose Estimation, Tracking, and Action Recognition on LieGroups. International Journal of Computer Vision (IJCV),123:454–478, 2017. 2

[72] Qi Ye, Shanxin Yuan, and Tae-Kyun Kim. Spatial AttentionDeep Net with Partial PSO for Hierarchical Hybrid HandPose Estimation. In European Conference on Computer Vi-sion (ECCV), pages 346–361, 2016. 2

[73] Shanxin Yuan, Qi Ye, Bjorn Stenger, Siddhant Jain, and Tae-Kyun Kim. Big Hand 2.2m Benchmark: Hand Pose Data Setand State of the Art Analysis. In The IEEE Conference onComputer Vision and Pattern Recognition (CVPR), 2017. 1

[74] Xingyi Zhou, Qingfu Wan, Wei Zhang, Xiangyang Xue, andYichen Wei. Model-based deep hand pose estimation. InThe International Joint Conference on Artificial Intelligence(IJCAI), pages 2421–2427, 2016. 5

[75] Christian Zimmermann and Thomas Brox. Learning to Esti-mate 3D Hand Pose from Single RGB Images. In The IEEEInternational Conference on Computer Vision (ICCV), pages4913–4921, 2017. 1, 2, 3, 8, 10

[76] Christian Zimmermann, Duygu Ceylan, Jimei Yang,Bryan C. Russell, Max J. Argus, and Thomas Brox. Frei-HAND: A Dataset for Markerless Capture of Hand Poseand Shape from Single RGB Images. In The IEEE Interna-tional Conference on Computer Vision (ICCV), pages 813–822, 2019. 1, 2, 4, 5, 7, 8