TSC-DL: Unsupervised trajectory segmentation of multi-modal surgical demonstrations with Deep Learning

TitleTSC-DL: Unsupervised trajectory segmentation of multi-modal surgical demonstrations with Deep Learning
Publication TypeConference Proceedings
Year of Publication2016
AuthorsMurali, A., Garg A., Krishnan S., Pokory F. T., Abbeel P., Darrell T., & Goldberg K.
Published inIEEE International Conference on Robotics and Automation (ICRA)
Date Published05/2016
ISBN Number978-1-4673-8026-3
Accession Number16055613
KeywordsFeature extraction, Hidden Markov models, Kinematics, machine learning, Motion segmentation, Visualization

The growth of robot-assisted minimally invasive surgery has led to sizable datasets of fixed-camera video and kinematic recordings of surgical subtasks. Segmentation of these trajectories into locally-similar contiguous sections can facilitate learning from demonstrations, skill assessment, and salvaging good segments from otherwise inconsistent demonstrations. Manual, or supervised, segmentation can be prone to error and impractical for large datasets. We present Transition State Clustering with Deep Learning (TSC-DL), a new unsupervised algorithm that leverages video and kinematic data for task-level segmentation, and finds regions of the visual feature space that correlate with transition events using features constructed from layers of pre-trained image classification Deep Convolutional Neural Networks (CNNs). We report results on three datasets comparing Deep Learning architectures (AlexNet and VGG), choice of convolutional layer, dimensionality reduction techniques, visual encoding, and the use of Scale Invariant Feature Transforms (SIFT). We find that the deep architectures extract features that result in up-to a 30.4% improvement in Silhouette Score (a measure of cluster tightness) over the traditional “shallow” features from SIFT. We also present cases where TSC-DL discovers human annotator omissions. Supplementary material, data and code is available at: http://berkeleyautomation.github.io/tsc-dl/.

ICSI Research Group