What you'll learn
Explain the three meanings of recursive in deep learning: tree networks, looped depth and Recursive Language Models
Build a small decoder-only transformer from scratch and use it as a measured baseline for every other architecture
Implement recursive networks, Tree-LSTM and a weight-tied looped transformer, and compare them at equal parameters
Build a Recursive Language Model loop by hand with a Python REPL and a local model, and read every step of its trace
Add sliding-window and linear attention to the baseline and measure memory, speed and quality against sequence length
Implement a selective state space layer, train a tiny Mamba-style model and compare pretrained Mamba with Pythia on CPU
Build mixture-of-experts layers, watch expert load with and without balancing, and inspect a real MoE router
Measure the KV cache, manage it with the transformers cache classes and keep generation stable with attention sinks
Build and evaluate a production-style RAG pipeline with chunking, FAISS, hybrid search, re-ranking and faithfulness checks
Run fair architecture comparisons and write an architecture selection report backed by a complete results table

Requirements
Working Python and basic PyTorch: you can read a training loop and run a script from the command line
A computer with a multi-core CPU. No GPU is needed: the course bench is an 8-CPU machine and every lab is measured on it

Description
"This course contains the use of artificial intelligence."

New architecture papers arrive every week, and most claims are made on hardware and at scales you will never reproduce. This course teaches you to understand the main ideas beyond the plain transformer, implement small versions of them yourself, and measure them fairly on an ordinary CPU, so you can decide what is worth using.

You start by building your own bench and lab environment: CPU PyTorch, Hugging Face transformers and a pinned set of dependencies. Then you build a tiny decoder-only transformer from scratch, train it, and record its speed, memory and quality in a results table. One benchmark harness is reused by every architecture that follows, so every row in the table is comparable.

Recursion is covered in all three of its meanings. You build recursive networks over trees and a Tree-LSTM, then weight-tied looped transformers, and look at Universal Transformers, Mixture-of-Recursions, HRM and the Tiny Recursive Model. The Recursive Language Model section builds an RLM loop by hand: the long prompt lives as a variable in a Python REPL, and the root model calls sub-models on pieces of it. You run it against a local model served by Ollama, read the full trace, try the official library, and see where a small root model fails.

The middle of the course works through the main efficiency ideas. Sparse, sliding-window and linear attention are implemented and timed; FlashAttention, MQA, GQA and latent attention are explained with honest notes on what needs a GPU. You implement a selective state space layer, train a tiny Mamba-style model, and compare pretrained Mamba and Pythia on the same data. RWKV, RetNet, Gated DeltaNet and today's hybrid stacks are explained, and an associative recall task shows why hybrids keep an attention layer.

Mixture of experts gets its own section: you build a router, watch expert load with and without the balancing loss, cause router collapse on purpose, and inspect the routing choices of a real small MoE model token by token. Memory follows, from external memory networks to the KV cache, which you measure, size from the model configuration and manage with the transformers cache classes and attention sinks.

Retrieval is treated as external memory you can update. You compare chunking strategies, choose a small embedding model and check its licence, build FAISS indexes, combine BM25 and dense search, add a cross-encoder re-ranker, and score retrieval with recall, MRR and nDCG on a labelled set. A complete local RAG pipeline answers with citations and a faithfulness check, and is compared with an RLM and a long prompt on the same questions.

The last sections are about decisions. You learn the rules of fair comparison, the pitfalls of perplexity, how to measure throughput and time to first token properly, and what a toy experiment can and cannot say about scaling. The capstone compares three architectures on one task with one harness and ends with a written architecture selection report. Eleven downloadable documents come with the course, including a paper-claim checklist, the fair comparison protocol, a CPU bench setup guide, KV cache sizing and MoE checklists, a RAG evaluation template and the selection report template. All numbers in the course are measured on the course bench and are presented as such.

Who this course is for
ML and LLM engineers who want to understand what comes after the plain transformer and when each design is worth it
Software engineers building LLM applications who need to choose between long context, RAG and recursive inference
Students and researchers who want hands-on, reproducible implementations instead of paper summaries