Jiakun Liu

EECS Department, University of California, Berkeley

Technical Report No. UCB/EECS-2022-134

May 17, 2022

http://www2.eecs.berkeley.edu/Pubs/TechRpts/2022/EECS-2022-134.pdf

Despite recent progress in reinforcement learning (RL), RL algorithms for exploration still remain an active area of research. Existing methods often focus on state-based metrics, which do not consider the underlying causal structures of the environment, and while re- cent research has begun to explore RL environments for causal learning, these environments primarily leverage causal information through causal inference or induction rather than ex- ploration. In contrast, human children—some of the most proficient explorers—have been shown to use causal information to great benefit. In this work, we introduce a novel RL environment designed with a controllable causal structure, which allows us to evaluate ex- ploration strategies used by both agents and children in a unified environment. In addition, through experimentation on both computation models and children, we demonstrate that there are significant differences between information-gain optimal RL exploration in causal environments and the exploration of children in the same environments. We conclude with a discussion of how these findings may inspire new directions of research into efficient explo- ration and disambiguation of causal structures for RL algorithms.

Advisors: John F. Canny

---

BibTeX citation:

@mastersthesis{Liu:EECS-2022-134,
    Author= {Liu, Jiakun},
    Title= {Learning Causal Overhypotheses through Exploration in Children and Computational Models},
    School= {EECS Department, University of California, Berkeley},
    Year= {2022},
    Month= {May},
    Url= {http://www2.eecs.berkeley.edu/Pubs/TechRpts/2022/EECS-2022-134.html},
    Number= {UCB/EECS-2022-134},
    Abstract= {Despite recent progress in reinforcement learning (RL), RL algorithms for exploration still
remain an active area of research. Existing methods often focus on state-based metrics,
which do not consider the underlying causal structures of the environment, and while re-
cent research has begun to explore RL environments for causal learning, these environments
primarily leverage causal information through causal inference or induction rather than ex-
ploration. In contrast, human children—some of the most proficient explorers—have been
shown to use causal information to great benefit. In this work, we introduce a novel RL
environment designed with a controllable causal structure, which allows us to evaluate ex-
ploration strategies used by both agents and children in a unified environment. In addition,
through experimentation on both computation models and children, we demonstrate that
there are significant differences between information-gain optimal RL exploration in causal
environments and the exploration of children in the same environments. We conclude with
a discussion of how these findings may inspire new directions of research into efficient explo-
ration and disambiguation of causal structures for RL algorithms.},
}

EndNote citation:

%0 Thesis
%A Liu, Jiakun 
%T Learning Causal Overhypotheses through Exploration in Children and Computational Models
%I EECS Department, University of California, Berkeley
%D 2022
%8 May 17
%@ UCB/EECS-2022-134
%U http://www2.eecs.berkeley.edu/Pubs/TechRpts/2022/EECS-2022-134.html
%F Liu:EECS-2022-134