by L Sun, U Klank and M Beetz
Abstract:
This paper investigates the inside-out recognition of everyday manipulation tasks using a gaze-directed camera, which is a camera that actively directs at the visual attention focus of the person wearing the camera. We present EYEWATCHME, an integrated vision and state estimation system that at the same time tracks the positions and the poses of the acting hands, the pose that the manipulated object, and the pose of the observing camera. Taken together, EYEWATCHME provides comprehensive data for learning predictive models of vision-guided manipulation that include the objects people are attending, the interaction of attention and reaching/grasping, and the segmentation of reaching and grasping using visual attention as evidence. Key technical contributions of this paper include an ego view hand tracking system that estimates 27 DOF hand poses. The hand tracking system is capable of detecting hands and estimating their poses despite substantial self-occlusion caused by the hand and occlusions caused by the manipulated object. EYEWATCHME can also cope with blurred images that are caused by rapid eye movements. The second key contribution is the of the integrated activity recognition system that simultaneously tracks the attention of the person, the hand poses, and the poses of the manipulated objects in terms of a global scene coordinates. We demonstrate the operation of EYEWATCHME in the context of kitchen tasks including filling a cup with water.
Reference:
EYEWATCHME - 3D Hand and object tracking for inside out activity analysis (L Sun, U Klank and M Beetz), In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2009. CVPR 2009., 2009.
Bibtex Entry:
@inproceedings{sun_eyewatchme_2009,
author = {L Sun and U Klank and M Beetz},
title = {{EYEWATCHME} - {3D} Hand and object tracking for inside out activity
analysis},
booktitle = {{IEEE} Computer Society Conference on Computer Vision and Pattern
Recognition, 2009. {CVPR} 2009.},
year = {2009},
pages = {9--16},
month = {jun},
abstract = {This paper investigates the inside-out recognition of everyday manipulation
tasks using a gaze-directed camera, which is a camera that actively
directs at the visual attention focus of the person wearing the camera.
We present {EYEWATCHME}, an integrated vision and state estimation
system that at the same time tracks the positions and the poses of
the acting hands, the pose that the manipulated object, and the pose
of the observing camera. Taken together, {EYEWATCHME} provides comprehensive
data for learning predictive models of vision-guided manipulation
that include the objects people are attending, the interaction of
attention and reaching/grasping, and the segmentation of reaching
and grasping using visual attention as evidence. Key technical contributions
of this paper include an ego view hand tracking system that estimates
27 {DOF} hand poses. The hand tracking system is capable of detecting
hands and estimating their poses despite substantial self-occlusion
caused by the hand and occlusions caused by the manipulated object.
{EYEWATCHME} can also cope with blurred images that are caused by
rapid eye movements. The second key contribution is the of the integrated
activity recognition system that simultaneously tracks the attention
of the person, the hand poses, and the poses of the manipulated objects
in terms of a global scene coordinates. We demonstrate the operation
of {EYEWATCHME} in the context of kitchen tasks including filling
a cup with water.},
doi = {10.1109/CVPR.2009.5204358},
keywords = {{3D} object tracking, blurred images, computer graphics, {EYEWATCHME},
gaze-directed camera, grasping segmentation, human computer interaction,
image restoration, image segmentation, image sensors, inside out
activity analysis, integrated activity recognition system, object
recognition, reaching segmentation, state estimation system, substantial
self-occlusion, {tracking3D} hand tracking, vision-guided manipulation},
}