Sequence Modeling

Recurrent Neural Network

• • • [RNN] D. Rumelhart, G. Hinton, R. Williams, "Learning Representations by Back-propagating Errors", Nature, 1986

    • [LSTM] S. Hochreiter, J. Schmidhuber, "Long Short-Term Memory", Neural Computation, 1997

    • [GRU] K. Cho, B. van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, Y. Bengio, "Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation", EMNLP, 2014   

Transformers

• • • [Transformer] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, Ł. Kaiser, I. Polosukhin, "Attention Is All You Need", NeurIPS, 2017

    • [Transformer-XL] Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. Le, R. Salakhutdinov, "Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context", ACL, 2019   

    • [Reformer] N. Kitaev, Ł. Kaiser, A. Levskaya, "Reformer: The Efficient Transformer", ICLR, 2020

    • [Longformer] I. Beltagy, M. Peters, A. Cohan, "Longformer: The Long-Document Transformer", arXiv, 2020

    • [Big Bird] M. Zaheer, G. Guruganesh, K. Dubey, J. Ainslie, C. Alberti, S. Ontanon, P. Pham, A. Ravula, Q. Wang, L. Yang, A. Ahmed, "Big Bird: Transformers for Longer Sequences", NeurIPS, 2020

State Space Models (SSMs)

Basic and Theory

• [Kalman Filter] R.E. Kalman, "A New Approach to Linear Filtering and Prediction Problems", Journal of Basic Engineering, 1960.

SSMs & Deep Learning:

• [Deep State Space Models] R. Krishnan, U. Shalit, D. Sontag, "Deep Kalman Filters", NeurIPS, 2015.

• [Structured Inference Networks] M. Fraccaro, S.K. Sønderby, U. Paquet, O. Winther, "Sequential Neural Models with Stochastic Layers", NeurIPS, 2016.

S4 Architecture

Core S4 Concepts

• • [HiPPO] A. Gu, T. Dao, S. Ermon, A. Rudra, C. Ré, "HiPPO: Recurrent Memory with Optimal Polynomial Projections", NeurIPS, 2020.

  • [S4] A. Gu, K. Goel, C. Ré, "Efficiently Modeling Long Sequences with Structured State Spaces", ICLR, 2021.

S4 Extensions

• • A. Gu, A. Gupta, J. Berant, "Structured State Spaces for Sequence Modeling", ICML, 2022.

  • [S4D] S. Patil, S. Srinivasan, A. Kannan, V. P. Gadde, A. Gu, "S4D: A Transformer-free Framework for Long-Range Sequence Modeling in Speech", INTERSPEECH, 2023

Mamba

• • [Mamba] A. Gu, T. Dao, "Mamba: Linear-Time Sequence Modeling with Selective State Spaces", arXiv, 2023