Sparse Autoencoders on Programmer.ie: Modern AI programming

Models From First Principles 06: Inside Tiny — Residual Blocks, Attention and Sparse Autoencoders

Sat, 08 Aug 2026 15:00:00 +0100

Inside Tiny: Residual Blocks, Attention and Sparse Autoencoders

In the previous post, we built a compact recursive model around one idea:

context + candidate + latent state
              ↓
          projection
              ↓
         reusable core
              ↓
       proposed update
              ↓
      z ← z + α · update
              ↓
            repeat

That architecture looked more sophisticated than MR.Q, EBT or SICQL because it introduced recurrence.

But the central idea of this series is that a model stops looking mysterious when we keep opening it.

Models From First Principles 05: Tiny — Recursive Reasoning With a Small Neural Network

Sat, 08 Aug 2026 14:55:00 +0100

Tiny — Recursive Reasoning With a Small Neural Network

The previous post introduced a much more ambitious architecture.

Instead of taking one representation and predicting from it once, the Hierarchical Reasoning Model repeatedly updated two latent states:

input
  ↓
low-level state
  ↓ ↓ ↓
high-level state
  ↓
repeat

That gave us something genuinely new:

computation could continue without adding a new set of parameters for every step.