105 min listen
BI 105 Sanjeev Arora: Off the Convex Path
FromBrain Inspired
ratings:
Length:
62 minutes
Released:
May 17, 2021
Format:
Podcast episode
Description
Sanjeev and I discuss some of the progress toward understanding how deep learning works, specially under previous assumptions it wouldn't or shouldn't work as well as it does. Deep learning theory poses a challenge for mathematics, because its methods aren't rooted in mathematical theory and therefore are a "black box" for math to open. We discuss how Sanjeev thinks optimization, the common framework for thinking of how deep nets learn, is the wrong approach. Instead, a promising alternative focuses on the learning trajectories that occur as a result of different learning algorithms. We discuss two examples of his research to illustrate this: creating deep nets with infinitely large layers (and the networks still find solutions among the infinite possible solutions!), and massively increasing the learning rate during training (the opposite of accepted wisdom, and yet, again, the network finds solutions!). We also discuss his past focus on computational complexity and how he doesn't share the current neuroscience optimism comparing brains to deep nets.
Sanjeev's website.His Research group website.His blog: Off The Convex Path.Papers we discussOn Exact Computation with an Infinitely Wide Neural Net.An Exponential Learning Rate Schedule for Deep LearningRelatedThe episode with Andrew Saxe covers related deep learning theory in episode 52.Omri Barak discusses the importance of learning trajectories to understand RNNs in episode 97.Sanjeev mentions Christos Papadimitriou.
Timestamps
0:00 - Intro
7:32 - Computational complexity
12:25 - Algorithms
13:45 - Deep learning vs. traditional optimization
17:01 - Evolving view of deep learning
18:33 - Reproducibility crisis in AI?
21:12 - Surprising effectiveness of deep learning
27:50 - "Optimization" isn't the right framework
30:08 - Infinitely wide nets
35:41 - Exponential learning rates
42:39 - Data as the next frontier
44:12 - Neuroscience and AI differences
47:13 - Focus on algorithms, architecture, and objective functions
55:50 - Advice for deep learning theorists
58:05 - Decoding minds
Sanjeev's website.His Research group website.His blog: Off The Convex Path.Papers we discussOn Exact Computation with an Infinitely Wide Neural Net.An Exponential Learning Rate Schedule for Deep LearningRelatedThe episode with Andrew Saxe covers related deep learning theory in episode 52.Omri Barak discusses the importance of learning trajectories to understand RNNs in episode 97.Sanjeev mentions Christos Papadimitriou.
Timestamps
0:00 - Intro
7:32 - Computational complexity
12:25 - Algorithms
13:45 - Deep learning vs. traditional optimization
17:01 - Evolving view of deep learning
18:33 - Reproducibility crisis in AI?
21:12 - Surprising effectiveness of deep learning
27:50 - "Optimization" isn't the right framework
30:08 - Infinitely wide nets
35:41 - Exponential learning rates
42:39 - Data as the next frontier
44:12 - Neuroscience and AI differences
47:13 - Focus on algorithms, architecture, and objective functions
55:50 - Advice for deep learning theorists
58:05 - Decoding minds
Released:
May 17, 2021
Format:
Podcast episode
Titles in the series (99)
BI 101 Steve Potter: Motivating Brains In and Out of Dishes: Steve and I discuss his book, How to Motivate Your Students to Love Learning, which is both a memoir and a guide for teachers and students to optimize the learning experience for intrinsic motivation. Steve taught neuroscience and engineering courses whi by Brain Inspired