Seeing to learn (S2l) is an computer vision-robotics project currently ongoing at University of Leeds, which is aimed at developing observation learning methods for robotics systems. It addresses the inability of current robotic systems to learn from human demonstrations. The project envisions a future where robots could acquire new skills by just observing humans perform a task or even by watching online tutorial videos of demonstrations. In future the robots equipped with these learning methods could be applied in several real world conditions ranging from home to work environments such as construction sites where it could learn to perform the relevant tasks of drilling holes, hammering nails or screwing a bolt just by observing other workers.
One shot observation learning using visual activity features[view paper]
Observation learning is the process of learning a task by observing an expert demonstrator. We present a robust observation learning method for robotic systems. Our principle contributions are in introducing a one shot learning method where only a single demonstration is needed for learning and in proposing a novel feature extraction method for extracting unique activity features from the demonstration. Reward values are then generated from these demonstrations. We use a learning algorithm with these rewards to learn the controls for a robotic manipulator to perform the demonstrated task. With simulation and real robot experiments, we show that the proposed method can be used to learn tasks from a single demonstration under varying conditions of viewpoints, object properties, morphology of manipulators and scene backgrounds.
Experiment 1: Reaching a target
Experiment 2: Pushing object to target
Publications Leo Pauly, Defining the problem of observation Learning, Technical report, 2017. [report]
Leo Pauly, Wisdom C. Agboh, Mohamed Abdellatif, David C. Hogg, Raul Fuentes “One-Shot Observation Learning”, IROS 2018 Workshop: Examining Sensing Modalities for Robust and Dextrous Object Manipulation, Poster, 2018.[poster]
The code and datasets being used in this project will be made available soon.