26 min listen
Language Modeling With State Space Models with Dan Fu - #630
FromThe TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)
Language Modeling With State Space Models with Dan Fu - #630
FromThe TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)
ratings:
Length:
28 minutes
Released:
May 22, 2023
Format:
Podcast episode
Description
Today we’re joined by Dan Fu, a PhD student at Stanford University. In our conversation with Dan, we discuss the limitations of state space models in language modeling and the search for alternative building blocks that can help increase context length without being computationally infeasible. Dan walks us through the H3 architecture and Flash Attention technique, which can reduce the memory footprint of a model and make it feasible to fine-tune. We also explore his work on improving language models using synthetic languages, the issue of long sequence length affecting both training and inference in models, and the hope for finding something sub-quadratic that can perform language processing more effectively than the brute force approach of attention.
The complete show notes for this episode can be found at https://twimlai.com/go/630
The complete show notes for this episode can be found at https://twimlai.com/go/630
Released:
May 22, 2023
Format:
Podcast episode
Titles in the series (100)
This Week in ML & AI - 6/24/16: Dueling Neural Networks at ICML, Plus Training a Robotic Housekeeper: This Week in Machine Learning & AI brings you the… by The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)