Can I use universe to record human demonstration and apply behavior clone before RL?


#1

The blog post says

We can use human performance as a meaningful baseline, and record human demonstrations by simply saving VNC traffic. We’ve found demonstrations to be extremely useful in initializing agents with sensible policies with behavioral cloning (i.e. use supervised learning to mimic what the human does), before switching to RL to optimize for the given reward function.

So, can I use universe to record my demonstration now?
If yes, how can I do this?
And how can I use the records?
Is there any documentation on this?


#2

We haven’t released the code for this, but will in upcoming weeks. If you’d like early access, please sign up here: https://docs.google.com/forms/d/e/1FAIpQLScAiW-kIS0mz6hdzzFZJJFlXlicDvQs1TX9XMEkipNwjV5VlA/viewform.


#3

Does anyone know of any papers exploring this in a recent context? Everything I’m finding is either very theoretical or untested against recently developed algorithms like DQN.


#4

Could you give some examples of these papers (even if they are not 100% related to this question)? I would be very interested in reading them. Thanks!


#5

Here are my notes/references so far:


Cooperative Inverse Reinforcement Learning

  • good source of other references
  • human and robot share a reward function
  • human knows reward function
  • robot does not know reward function

good description of above: http://pemami4911.github.io/paper-summaries/2016/08/11/Coop-Inverse-RL.html


feature engineered version:

  • goal is motion through space of robot arm
  • motion is defined by a linear set of points in state-space
  • clusters of points in state-space by human action is used to guide policy
  • tasks: scooping and pouring of coffee beans, placing item
    • quality of completed task is measured by weight of scooped or poured beans

http://www.mcgovern-fagg.org/amy/courses/cs5973_fall2005/lfd.pdf
Robot Learning From Demonstration

pendulum swing up task

  • uses video of human hand to assist in learning action to take by robot arm
  • many issues stemming from the physical differences between the robot arm
    and human arm
    • can’t replicate motion exactly
    • human deviated from 2d plane assumed in robot case
    • different grip

http://www.scholarpedia.org/article/Robot_learning_by_demonstration

high level overview, pretty good, I think, something about it unsatisfying
though. only motor control tasks, no games


cart pole robot arm:


Inverse Reinforcement Learning (slides)
high level overview, good foundations


#6

Hi,
Any ETA on when the code is gonna be available on how to train agents from recorded human demos?
Cheers.


#7

Any leads on this?