Allan Jabri

EECS Department, University of California, Berkeley

Technical Report No. UCB/EECS-2023-192

July 7, 2023

http://www2.eecs.berkeley.edu/Pubs/TechRpts/2023/EECS-2023-192.pdf

Any useful agent will face many tasks and must rely on transfer of prior knowledge acquired in a scalable manner. This thesis explores inductive biases that enable scalable pre-training of representations -- and algorithms that bind them -- from the design of architectures capable of adaptive computation for scalable generative modeling, to self-supervised objectives that prepare embodied agents with mechanisms for state representation and reward maximization.

First, I consider the challenge of gracefully scaling generative models to high-dimensional data, motivating the importance of adaptive computation, a property missing from predominant architectures. This leads to a simple attention-based architecture for diffusion models capable of dedicating computation adaptively across its input and output, attaining superior performance in image and video generation despite being more domain-agnostic and efficient. Attention visualizations demonstrate how the model learns to allocate computation to more complex parts of samples, and in cases of high redundancy such as video prediction, can even copy information when needed.

Next, I show how self-supervised objectives that exploit more domain knowledge can be used to efficiently solve related downstream tasks. In the domain of perception, I show how a simple self-supervised objective for space-time attention can be used to solve a range of tasks involving temporal correspondence, a central challenge in state representation for embodied agents. In the domain of reinforcement learning, I motivate the importance of scalable construction of task distributions and demonstrate how meta-reinforcement learners can be pre-trained with self-supervised reward models.

Finally, I conclude with a perspective on open problems in scalable pre-training, with a focus on the interplay between transfer across modalities, universal generative modeling objectives for discrete and continuous data, and adaptive computation.

Advisors: Alexei (Alyosha) Efros

---

BibTeX citation:

@phdthesis{Jabri:EECS-2023-192,
    Author= {Jabri, Allan},
    Title= {Scalable Binding},
    School= {EECS Department, University of California, Berkeley},
    Year= {2023},
    Month= {Jul},
    Url= {http://www2.eecs.berkeley.edu/Pubs/TechRpts/2023/EECS-2023-192.html},
    Number= {UCB/EECS-2023-192},
    Abstract= {Any useful agent will face many tasks and must rely on transfer of prior knowledge acquired in a scalable manner. This thesis explores inductive biases that enable scalable pre-training of representations -- and algorithms that bind them -- from the design of architectures capable of adaptive computation for scalable generative modeling, to self-supervised objectives that prepare embodied agents with mechanisms for state representation and reward maximization.

First, I consider the challenge of gracefully scaling generative models to high-dimensional data, motivating the importance of adaptive computation, a property missing from predominant architectures. This leads to a simple attention-based architecture for diffusion models capable of dedicating computation adaptively across its input and output, attaining superior performance in image and video generation despite being more domain-agnostic and efficient. Attention visualizations demonstrate how the model learns to allocate computation to more complex parts of samples, and in cases of high redundancy such as video prediction, can even copy information when needed.

Next, I show how self-supervised objectives that exploit more domain knowledge can be used to efficiently solve related downstream tasks. In the domain of perception, I show how a simple self-supervised objective for space-time attention can be used to solve a range of tasks involving temporal correspondence, a central challenge in state representation for embodied agents. In the domain of reinforcement learning, I motivate the importance of scalable construction of task distributions and demonstrate how meta-reinforcement learners can be pre-trained with self-supervised reward models.

Finally, I conclude with a perspective on open problems in scalable pre-training, with a focus on the interplay between transfer across modalities, universal generative modeling objectives for discrete and continuous data, and adaptive computation.},
}

EndNote citation:

%0 Thesis
%A Jabri, Allan 
%T Scalable Binding
%I EECS Department, University of California, Berkeley
%D 2023
%8 July 7
%@ UCB/EECS-2023-192
%U http://www2.eecs.berkeley.edu/Pubs/TechRpts/2023/EECS-2023-192.html
%F Jabri:EECS-2023-192