Tag: vanishing-gradients

All the articles with the tag "vanishing-gradients".

• Sequences Before Attention: RNNs, Vanishing Gradients, and Why Transformers Won

  Updated:19 Oct, 2026

  A recurrent network compresses everything it has read into one fixed-size vector, updated one step at a time. Worked by hand and backpropagated through time, then the two problems that sentence contains: gradients that vanish over distance (measured for RNN, GRU and LSTM), and a fixed bottleneck that breaks sequence-to-sequence models (measured on a reversal task, with and without attention). Plus a character-level RNN trained on this blog, the baseline the tiny GPT trained later in the series has to beat.