Saagar Sanghavi

EECS Department, University of California, Berkeley

Technical Report No. UCB/EECS-2023-287

December 15, 2023

http://www2.eecs.berkeley.edu/Pubs/TechRpts/2023/EECS-2023-287.pdf

Large Language Models (LLMs) have shown to be highly effective at performing in-context learning, where, given a prompt, the model can learn from the prompt and complete the sequence without needing to perform additional gradient steps or fine-tuning. In this project, we investigated the ability of Transformer models to perform in-context learning on linear dynamical systems. We first experimented with Transformers trained on a single system, where the task for evaluation was to filter noise on trajectories sampled from the same system. Then, we experimented with Transformers trained on multiple systems of the same type, where the task was to perform simultaneous system identification and filtering. This is still very much a work in progress, and I hope to continue to work on this in the coming weeks.

Advisors: Robert Full

---

BibTeX citation:

@mastersthesis{Sanghavi:EECS-2023-287,
    Author= {Sanghavi, Saagar},
    Title= {Transformers on Dynamical Systems - An Exploration of In-context Learning},
    School= {EECS Department, University of California, Berkeley},
    Year= {2023},
    Month= {Dec},
    Url= {http://www2.eecs.berkeley.edu/Pubs/TechRpts/2023/EECS-2023-287.html},
    Number= {UCB/EECS-2023-287},
    Abstract= {Large Language Models (LLMs) have shown to be highly effective at performing in-context learning,
where, given a prompt, the model can learn from the prompt and complete the sequence without needing
to perform additional gradient steps or fine-tuning. In this project, we investigated the ability of Transformer
models to perform in-context learning on linear dynamical systems. We first experimented with
Transformers trained on a single system, where the task for evaluation was to filter noise on trajectories
sampled from the same system. Then, we experimented with Transformers trained on multiple systems
of the same type, where the task was to perform simultaneous system identification and filtering. This
is still very much a work in progress, and I hope to continue to work on this in the coming weeks.},
}

EndNote citation:

%0 Thesis
%A Sanghavi, Saagar 
%T Transformers on Dynamical Systems - An Exploration of In-context Learning
%I EECS Department, University of California, Berkeley
%D 2023
%8 December 15
%@ UCB/EECS-2023-287
%U http://www2.eecs.berkeley.edu/Pubs/TechRpts/2023/EECS-2023-287.html
%F Sanghavi:EECS-2023-287