Ruishan Liu (PhD candidate) has received the Best Poster Award at the Bay Area Machine Learning Symposium, October 19, 2017. Ruishan belongs to the Stanford Laboratory for Machine Learning group, advised by Professor James Zou. Ruishan develops algorithms and theories in machine learning and reinforcement learning. She is also interested in applications in genomics and healthcare.
"The Effects of Memory Replay in Reinforcement Learning"
Experience replay is a key technique behind many recent advances in deep reinforcement learning. Despite its wide-spread application, very little is understood about the properties of experience replay. How does the amount of memory kept affect learning dynamics? Does it help to prioritize certain experiences?
In our work, we address these questions by formulating a dynamical systems ODE model of Q-learning with experience replay. We derive analytic solutions of the ODE for a simple setting. We show that even in this very simple setting, the amount of memory kept can substantially affect the agent's performance. Too much or too little memory both slow down learning.
We also proposed a simple algorithm for adaptively changing the memory buffer size which achieves consistently good empirical performance.
Congratulations to Ruishan!