Computer Science Thesis Oral
— 2:00pm
Location:
In Person and Virtual - ET
-
Traffic21 Classroom, Gates Hillman 6501 and Zoom
Speaker:
YUE (SOPHIE) GUO
,
Ph.D. Candidate, Computer Science Department, Carnegie Mellon University
http://sophieyueguo.com/
Enhancing Policy Transfer in Action Advising for Reinforcement Learning
As humans benefit from teachers' advice to accelerate learning, so could agents benefit from advice. Agents can not only learn from humans, such as via human expert demonstrations, but also from other agents. This thesis focuses on action advising, a knowledge transfer technique built upon the teacher-student paradigm of reinforcement learning. In this approach, the teacher agent provides action advice calculated from its policy given the student's observations. Although action advising has been studied over the past decade, the focus of the related work has primarily been on when to advise.
We extend the current state-of-the-art by studying the following additional challenges: (1) In existing work, advice is mostly given without explaining the rationale behind it, and thus does not provide an explanation for the generated advice; (2) Most of the work does not consider that the teacher might be suboptimal in a new environment or enable the student to discern when advice might not be applicable; (3) Very few studies enable the teacher to evaluate the quality of its advice before giving it to the student; (4) The teacher often has limited knowledge and may not examine the student's environment in detail, particularly the rewards received in a new environment; (5) The teacher is generally assumed to have a fixed pre-trained policy, and might not be able to provide flexible advice; (6) The advice from an agent is primarily aimed at improving another agent, while advising human is less common.
We present our solutions to tackle the aforementioned challenges and exhibit the empirical effectiveness of our proposed methods. We also propose potential pathways for subsequent research. Our ultimate goal is to delve into the intricacies and potentials of action advising in reinforcement learning, thereby directing future advancements in the field. Additionally, we broaden the scope by examining further applications of transfer learning, showcasing its utility and adaptability in varied contexts.
Thesis Committee:
Katia Sycara (Chair)
Fei Fang
Zico Kolter
Matthew E. Taylor (University of Alberta)
In Person and Zoom Participation. See announcement.