Posts
All the articles I've posted.
Sequences Before Attention: RNNs, Vanishing Gradients, and Why Transformers Won
Updated:A recurrent network compresses everything it has read into one fixed-size vector, updated one step at a time. Worked by hand and backpropagated through time, then the two problems that sentence contains: gradients that vanish over distance (measured for RNN, GRU and LSTM), and a fixed bottleneck that breaks sequence-to-sequence models (measured on a reversal task, with and without attention). Plus a character-level RNN trained on this blog, the baseline the tiny GPT trained later in the series has to beat.
Where Am I? Position in Attention, from Sinusoids to RoPE
Updated:Attention on its own can't tell word order apart, proved in two lines. How position gets in: added sinusoids, learned embeddings, ALiBi's distance penalty, and RoPE's rotation, derived so the query-key score depends only on relative distance. Then a tiny Transformer trained with each, tested on sequences four times longer than it trained on, on one task that needs exact relative position and one that needs to look back over any distance, and what the results say about long-context extension.
Semantic and Hybrid Search: BM25, Dense Retrieval and Rank Fusion
Updated:Keyword search matches the words a query uses; dense retrieval matches what it means. BM25 derived and worked on this blog's sections, dense retrieval and its truncation, reciprocal rank fusion, and the metrics. Then a benchmark built from the blog itself, English and Vietnamese: fusion helps in English and hurts in Vietnamese, diacritics matter, and technical Vietnamese can be searched with English keywords, with bootstrap intervals.
Serving Metrics and Prefill–Decode Disaggregation
Updated:Time to first token is a prefill problem and time per output token is a decode problem; serving both on one GPU makes them interfere, and splitting them trades that interference for a KV-cache transfer and a ratio to get right. The metrics and SLOs, goodput, chunked prefill, and disaggregation, then a step-by-step simulation of an 8B model on H100s under Poisson traffic, measuring every gap between tokens: why averages hide stalls, why chunked prefill sized to the target wins in an idealized model, and what disaggregation offers that the model leaves out.