Upcoming Seminars

Other UCLA departments frequently hold seminars related to Statistics and of likely of interest to our members. Here is the link to UCLA Biostatistics seminars: https://www.biostat.ucla.edu/events

How to Subscribe to the UCLA Statistics Seminars Mailing List

Join the UCLA Statistics seminars mailing list by sending an email to sympa@sympa.it.ucla.edu with “subscribe stat_seminars” (without quotation marks) in the subject field and the message body blank. This needs to be done from the address that is to be subscribed. After doing that please respond to the email that you receive. An automated email will be sent which confirms that you have been added.

How to Unsubscribe from the UCLA Statistics Seminars Mailing List

You may be receiving our seminar emails because you are directly subscribed to our seminars mailing list (or you may be one of our graduate students, undergraduate students, faculty, etc. and are subscribed to a different mailing list that also receives the seminar emails). If you are the former then you may unsubscribe from the seminar mailing list by sending an email to sympa@sympa.it.ucla.edu with “unsubscribe stat_seminars” (without quotation marks) in the subject field and the message body blank. This needs to be done from the address that is subscribed. After sending that email please follow the directions in the email response that you receive.

Viewing our Seminars Remotely

When viewing one of our live seminars remotely, it is optimal to have your Zoom settings such that you are using “Side-by-side: Speaker View”. You can see details of how to do this here.

Monday, 06/05/2023, Time: 11:00am – 12:15pm PT
Transformers As Statisticians: Provable In-Context Learning With In-Context Algorithm Selection

Song Mei, Assistant Professor
Departments of Statistics and Electrical Engineering and Computer Sciences, UC Berkeley

Broad 2100A

Abstract:

Neural sequence models based on the transformer architecture have demonstrated remarkable in-context learning (ICL) abilities, where they can perform new tasks when prompted with training and test examples, without any parameter update to the model. In this talk, we theoretically investigate the strong ICL abilities of transformers. We first provide a statistical theory for transformers to perform ICL by deriving end-to-end quantitative results for the expressive power, in-context prediction power, and sample complexity of pre-training. Concretely, we show that transformers can implement a broad class of standard machine learning algorithms in context, such as least squares, ridge regression, Lasso, convex risk minimization for generalized linear models (such as logistic regression), and gradient descent on two-layer neural networks, with near-optimal predictive power on various in-context data distributions. Building on these “base” ICL algorithms, intriguingly, we show that transformers can implement more complex ICL procedures involving in-context algorithm selection, akin to what a statistician can do in real life—A single transformer can adaptively select different base ICL algorithms—or even perform qualitatively different tasks—on different input sequences, without any explicit prompting of the right algorithm or task.

Bio:

Song Mei is an assistant professor of statistics at UC Berkeley. He received his Ph. D. from Stanford in June 2020. Song’s research lies at the intersection of statistics and machine learning. His recent research interests include high dimensional statistical inference, theory of deep learning, and theory of reinforcement learning.