Seeing to learn (S2l) is an computer vision-robotics project currently ongoing at University of Leeds, which is aimed at developing observation learning methods for robotics systems. It addresses the inability of current robotic systems to learn from human demonstrations. The project envisions a future where robots could acquire new skills by just observing humans perform a task or even by watching online tutorial videos of demonstrations. In future the robots equipped with these learning methods could be applied in several real world conditions ranging from home to work environments such as construction sites where it could learn to perform the relevant tasks of drilling holes, hammering nails or screwing a bolt just by observing other workers.

One shot observation learning using visual activity features

[view paper]

Abstract:

Observation learning is the process of learning a task by observing an expert demonstrator. We present a robust observation learning method for robotic systems. Our principle contributions are in introducing a one shot learning method where only a single demonstration is needed for learning and in proposing a novel feature extraction method for extracting unique activity features from the demonstration. Reward values are then generated from these demonstrations. We use a learning algorithm with these rewards to learn the controls for a robotic manipulator to perform the demonstrated task. With simulation and real robot experiments, we show that the proposed method can be used to learn tasks from a single demonstration under varying conditions of viewpoints, object properties, morphology of manipulators and scene backgrounds.

Experiment 1: Reaching a target

@leopauly
Demo
@leopauly
Episode 1
@leopauly
Episode 115

Experiment 2: Pushing object to target

    Demonstration Execution             Demonstration Execution
   
Episode 180
Episode 180
           
Episode 180
Episode 180
    Demonstration Execution             Demonstration Execution
   
Episode 180
Episode 180
           
Episode 180
Episode 180
    Demonstration Execution            
   
Episode 180
Episode 180
           

Publications

[1] Leo Pauly, Defining the problem of observation Learning, Technical report, 2017. [report]
[2]Leo Pauly, Wisdom C. Agboh, Mohamed Abdellatif, David C. Hogg, Raul Fuentes “One-Shot Observation Learning”, IROS 2018 Workshop: Examining Sensing Modalities for Robust and Dextrous Object Manipulation, Poster, 2018.[poster]

Team

@leopauly
Leo Pauly
(2017-present)
PhD Student
Univeristy of Leeds
@leopauly
Mohammad Kaykanloo
(Nov - Dec 2017)
Msc student
Univeristy of Leeds
@Prof.David Hogg
Prof.David Hogg
Professor of Artificial Intelligence
Univeristy of Leeds
@Dr.Raul Fuentes
Dr.Raul Fuentes
Associate Professor
Univeristy of Leeds

Downloads

The code and datasets being used in this project will be made available soon.