Posts
All the articles I've posted.
Agent Memory: Write, Read, Update, Forget
Updated:An agent's context is its working memory; everything else is storage with a retrieval policy. A taxonomy of agent memory, the four operations and the rule each needs, Generative Agents' recency, importance and relevance score worked, MemGPT's tiers, forgetting, memory as files, and privacy. Then retrieval policies measured on 250 conversations whose facts change, with ablations: how often each shows the model an outdated fact, and what fixing that costs.
Attention Alternatives: Sparse, Linear, and Recurrent
Updated:Softmax attention costs quadratic compute at prefill and a cache that grows with every token. There are three ways out: attend to fewer tokens, drop the softmax, or replace attention with a recurrence. Each is derived here and checked in code: linear attention matches the quadratic form to 1e-16 and runs 75x faster than it at 16K tokens on a laptop GPU, a 64-wide linear-attention state reads back 100% of 64 stored pairs but 35% of 512, and the configs of thirteen released models show what shipped.
What "Generative" Means: Autoregression, the Chain Rule, and Perplexity
Updated:A language model is one factorization, p(text) = the product of each token's probability given the ones before it, and training, sampling, caching and evaluation all follow from it. The chain rule worked on this blog, teacher forcing vs generation, what a small model writes when greedy and when sampling, perplexity derived from cross-entropy, why perplexity can't be compared across tokenizers and bits per byte can, measured with a character model and a BPE model on the same text.
Backpropagation, by Hand: The Chain Rule, Run Backward
Updated:Why a hidden layer can learn what a straight line can't, and how backpropagation gets the gradient of every weight in one backward pass. Worked by hand on XOR with every number, then as matrix rules for a whole layer, checked against finite differences, and costed: where the '6 × parameters FLOPs per training token' rule comes from.