× L residual stream input token Embed Unembed next-token probabilities h0 hl−1 hl hL ⋮ ⋮
h′ = hl−1 + Attn(Norm(hl−1))
hl = h′ + MLP(Norm(h′))

Figure 1. The residual-stream view of a decoder-only transformer (one token position shown). Each of the L layers has two sublayers, attention and MLP. Each sublayer reads from the stream, computes an update, and adds it back (⊕). Attention is the only part that also reads from other token positions. The activation vectors h0, …, hL are what we analyze and steer. Details such as the exact placement of normalization layers differ between models and are omitted.